-- Logs begin at Sun 2022-08-28 00:10:21 UTC, end at Sun 2022-08-28 01:52:07 UTC. -- Aug 28 00:40:23.935147 dv-cinder-master-3-iscsi systemd[1]: Starting Devstack devstack@keystone.service... Aug 28 00:40:23.946002 dv-cinder-master-3-iscsi devstack@keystone.service[81493]: [uWSGI] getting INI configuration from /etc/keystone/keystone-uwsgi-public.ini Aug 28 00:40:23.948058 dv-cinder-master-3-iscsi devstack@keystone.service[81493]: *** Starting uWSGI 2.0.18-debian (64bit) on [Sun Aug 28 00:40:23 2022] *** Aug 28 00:40:23.948058 dv-cinder-master-3-iscsi devstack@keystone.service[81493]: compiled with version: 10.0.1 20200405 (experimental) [master revision 0be9efad938:fcb98e4978a:705510a708d3642c9c962beb663c476167e4e8a4] on 11 April 2020 11:15:55 Aug 28 00:40:23.948058 dv-cinder-master-3-iscsi devstack@keystone.service[81493]: os: Linux-5.4.0-72-generic #80-Ubuntu SMP Mon Apr 12 17:35:00 UTC 2021 Aug 28 00:40:23.948058 dv-cinder-master-3-iscsi devstack@keystone.service[81493]: nodename: dv-cinder-master-3-iscsi Aug 28 00:40:23.948058 dv-cinder-master-3-iscsi devstack@keystone.service[81493]: machine: x86_64 Aug 28 00:40:23.948058 dv-cinder-master-3-iscsi devstack@keystone.service[81493]: clock source: unix Aug 28 00:40:23.948058 dv-cinder-master-3-iscsi devstack@keystone.service[81493]: pcre jit disabled Aug 28 00:40:23.948058 dv-cinder-master-3-iscsi devstack@keystone.service[81493]: detected number of CPU cores: 4 Aug 28 00:40:23.948058 dv-cinder-master-3-iscsi devstack@keystone.service[81493]: current working directory: / Aug 28 00:40:23.948058 dv-cinder-master-3-iscsi devstack@keystone.service[81493]: detected binary path: /usr/bin/uwsgi-core Aug 28 00:40:23.948058 dv-cinder-master-3-iscsi devstack@keystone.service[81493]: your processes number limit is 47680 Aug 28 00:40:23.948624 dv-cinder-master-3-iscsi devstack@keystone.service[81493]: your memory page size is 4096 bytes Aug 28 00:40:23.948624 dv-cinder-master-3-iscsi devstack@keystone.service[81493]: detected max file descriptor number: 2048 Aug 28 00:40:23.948624 dv-cinder-master-3-iscsi devstack@keystone.service[81493]: lock engine: pthread robust mutexes Aug 28 00:40:23.948624 dv-cinder-master-3-iscsi devstack@keystone.service[81493]: thunder lock: enabled Aug 28 00:40:23.948624 dv-cinder-master-3-iscsi devstack@keystone.service[81493]: uwsgi socket 0 bound to UNIX address /var/run/uwsgi/keystone-wsgi-public.socket fd 4 Aug 28 00:40:23.948624 dv-cinder-master-3-iscsi devstack@keystone.service[81493]: Python version: 3.8.10 (default, Jun 22 2022, 20:18:18) [GCC 9.4.0] Aug 28 00:40:23.987347 dv-cinder-master-3-iscsi devstack@keystone.service[81493]: Python main interpreter initialized at 0x56283b1cf9c0 Aug 28 00:40:23.987347 dv-cinder-master-3-iscsi devstack@keystone.service[81493]: python threads support enabled Aug 28 00:40:23.987347 dv-cinder-master-3-iscsi devstack@keystone.service[81493]: your server socket listen backlog is limited to 100 connections Aug 28 00:40:23.987347 dv-cinder-master-3-iscsi devstack@keystone.service[81493]: your mercy for graceful operations on workers is 90 seconds Aug 28 00:40:23.987751 dv-cinder-master-3-iscsi devstack@keystone.service[81493]: mapped 403077 bytes (393 KB) for 2 cores Aug 28 00:40:23.987832 dv-cinder-master-3-iscsi devstack@keystone.service[81493]: *** Operational MODE: preforking *** Aug 28 00:40:23.988063 dv-cinder-master-3-iscsi devstack@keystone.service[81493]: *** uWSGI is running in multiple interpreter mode *** Aug 28 00:40:23.988063 dv-cinder-master-3-iscsi devstack@keystone.service[81493]: spawned uWSGI master process (pid: 81493) Aug 28 00:40:23.988275 dv-cinder-master-3-iscsi systemd[1]: Started Devstack devstack@keystone.service. Aug 28 00:40:23.989286 dv-cinder-master-3-iscsi devstack@keystone.service[81493]: spawned uWSGI worker 1 (pid: 81507, cores: 1) Aug 28 00:40:23.989286 dv-cinder-master-3-iscsi devstack@keystone.service[81493]: spawned uWSGI worker 2 (pid: 81508, cores: 1) Aug 28 00:40:23.989443 dv-cinder-master-3-iscsi devstack@keystone.service[81493]: running "unix_signal:15 gracefully_kill_them_all" (master-start)... Aug 28 00:40:28.224726 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Aug 28 00:40:28.224726 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: from cryptography.utils import int_from_bytes Aug 28 00:40:28.225603 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Aug 28 00:40:28.225603 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: from cryptography.utils import int_from_bytes Aug 28 00:40:28.225714 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Aug 28 00:40:28.225714 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: from cryptography.utils import int_from_bytes Aug 28 00:40:28.226112 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Aug 28 00:40:28.226112 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: from cryptography.utils import int_from_bytes Aug 28 00:40:28.274441 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Aug 28 00:40:28.274676 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Aug 28 00:40:28.274759 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] command line args: [] {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Aug 28 00:40:28.275026 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Aug 28 00:40:28.275229 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] ================================================================================ {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Aug 28 00:40:28.275470 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] admin_token = **** {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 28 00:40:28.275603 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Aug 28 00:40:28.275681 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] config_dir = [] {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 28 00:40:28.275809 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Aug 28 00:40:28.275941 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 28 00:40:28.276030 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] command line args: [] {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Aug 28 00:40:28.276117 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] config_source = [] {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 28 00:40:28.276190 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] debug = True {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 28 00:40:28.276255 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Aug 28 00:40:28.276439 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 28 00:40:28.276624 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] ================================================================================ {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Aug 28 00:40:28.276624 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] admin_token = **** {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 28 00:40:28.276747 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 28 00:40:28.276922 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] config_dir = [] {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 28 00:40:28.277044 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] insecure_debug = False {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 28 00:40:28.277113 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 28 00:40:28.277172 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 28 00:40:28.277239 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] config_source = [] {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 28 00:40:28.277416 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 28 00:40:28.277504 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] debug = True {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 28 00:40:28.277638 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 28 00:40:28.277745 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] list_limit = None {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 28 00:40:28.277817 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] log_config_append = None {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 28 00:40:28.277875 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 28 00:40:28.277937 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 28 00:40:28.278000 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] insecure_debug = False {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 28 00:40:28.278181 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] log_dir = None {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 28 00:40:28.278266 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 28 00:40:28.278336 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] log_file = None {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 28 00:40:28.278476 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 28 00:40:28.278554 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 28 00:40:28.278742 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] list_limit = None {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 28 00:40:28.278866 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 28 00:40:28.278940 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] log_config_append = None {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 28 00:40:28.279016 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 28 00:40:28.279080 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 28 00:40:28.279258 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 28 00:40:28.279395 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] log_dir = None {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 28 00:40:28.279461 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 28 00:40:28.279535 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] log_file = None {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 28 00:40:28.279595 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 28 00:40:28.279659 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 28 00:40:28.279721 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 28 00:40:28.279832 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 28 00:40:28.279978 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 28 00:40:28.280073 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 28 00:40:28.280169 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 28 00:40:28.280290 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 28 00:40:28.280383 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 28 00:40:28.280455 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 28 00:40:28.280624 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 28 00:40:28.280716 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 28 00:40:28.280884 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 28 00:40:28.281024 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 28 00:40:28.281116 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 28 00:40:28.281180 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 28 00:40:28.281357 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] notification_format = cadf {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 28 00:40:28.281458 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 28 00:40:28.281593 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 28 00:40:28.281698 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 28 00:40:28.281826 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 28 00:40:28.282069 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 28 00:40:28.282180 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.42/identity {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 28 00:40:28.282296 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] publish_errors = False {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 28 00:40:28.282395 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 28 00:40:28.282488 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 28 00:40:28.282569 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] notification_format = cadf {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 28 00:40:28.282714 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 28 00:40:28.282794 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 28 00:40:28.282907 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 28 00:40:28.283114 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 28 00:40:28.283283 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.42/identity {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 28 00:40:28.283368 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 28 00:40:28.283466 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] publish_errors = False {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 28 00:40:28.283647 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] standard_threads = False {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 28 00:40:28.283721 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 28 00:40:28.283850 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] strict_password_check = False {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 28 00:40:28.283934 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 28 00:40:28.284046 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 28 00:40:28.284126 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 28 00:40:28.284202 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] use_eventlog = False {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 28 00:40:28.284310 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 28 00:40:28.284389 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] use_journal = False {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 28 00:40:28.284498 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] use_json = False {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 28 00:40:28.284573 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 28 00:40:28.284673 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] use_stderr = False {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 28 00:40:28.284749 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] standard_threads = False {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 28 00:40:28.284889 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] use_syslog = False {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 28 00:40:28.284984 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] strict_password_check = False {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 28 00:40:28.285069 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] watch_log_file = False {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 28 00:40:28.285242 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 28 00:40:28.285319 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.285436 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] use_eventlog = False {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 28 00:40:28.285525 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.285603 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] use_journal = False {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 28 00:40:28.285760 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] use_json = False {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 28 00:40:28.285836 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.285934 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] use_stderr = False {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 28 00:40:28.286013 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.286092 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] use_syslog = False {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 28 00:40:28.286229 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.286312 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] watch_log_file = False {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Aug 28 00:40:28.286461 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.286570 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.286703 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.286810 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.286932 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.287071 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.287171 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.287309 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.287411 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.287558 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.287660 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.287794 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.287889 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.288042 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.288123 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.288278 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.288354 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.288524 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.288605 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.288770 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.288851 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.289039 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.289136 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.289276 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.289353 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.289521 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.289605 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.289756 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.289835 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.289983 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.290059 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.290231 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.290387 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.290461 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.290536 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.290705 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.290782 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.290951 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.291034 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.291188 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.291281 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.291391 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.291467 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.291614 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.291702 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.291845 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.291928 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.292088 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.292170 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.292331 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.292415 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.292555 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.292631 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.292780 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.292855 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.293041 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.293136 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.293278 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.293354 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.293507 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.293582 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.293749 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.293832 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.293978 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.294052 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.294207 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.294282 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.294436 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.294522 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.294660 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.294740 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.294886 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.294962 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.295110 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.295194 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.295333 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.295410 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.295570 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.295645 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.295813 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.295965 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.296044 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.296114 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.296251 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.296327 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.296474 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.296567 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.296702 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.296777 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.296927 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.297032 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.297245 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.297335 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.297470 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.297553 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.297702 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.297778 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.297930 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.298032 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.298158 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.298264 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.298388 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.298498 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.298608 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.298704 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.298829 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.298931 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.299073 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.299160 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.299294 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.299373 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.299530 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.299609 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.299755 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.299840 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.299990 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.300068 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.300220 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.300300 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.300442 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.300527 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.300674 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.300758 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.300905 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.301035 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.301162 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.301252 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.301414 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.301496 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.301648 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.301726 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.301879 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.301958 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.302109 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.302187 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.302339 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.302423 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.302567 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.302648 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.302811 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] cache.enabled = True {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.302925 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.303075 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.303204 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.303324 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.303444 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.303562 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.303685 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.303812 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.303936 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.304058 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] cache.memcache_password = {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.304191 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.304316 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.304444 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] cache.enabled = True {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.304574 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.304717 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.304837 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.304917 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.305020 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.305094 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.305179 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] cache.memcache_sasl_enabled = False {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.305253 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.305322 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.305392 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] cache.memcache_password = {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.305514 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.305614 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.305757 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] cache.memcache_username = {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.305855 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.305992 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.306094 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.306224 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.306331 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.306458 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.306568 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] cache.memcache_sasl_enabled = False {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.306690 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.306831 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.306908 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.307072 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.307228 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.307324 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] cache.memcache_username = {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.307400 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.307563 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.307638 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.307787 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.307873 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.308014 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.308092 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.308238 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.308325 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.308473 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.308549 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.308703 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.308822 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.308934 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.309073 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.309187 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.309264 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.309418 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.309497 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.309654 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.309732 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.309887 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.309970 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.310121 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] auth.external = None {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.310199 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.310347 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] auth.mapped = None {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.310430 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.310593 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.310670 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.310832 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.310922 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.311065 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] auth.password = None {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.311144 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.311288 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] auth.token = None {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.311366 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.311521 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.311599 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.311752 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] catalog.caching = True {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.311830 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] auth.external = None {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.311991 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.312069 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] auth.mapped = None {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.312217 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.312294 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.312458 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.312537 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.312719 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.312797 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] auth.password = None {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.312947 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.313050 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] auth.token = None {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.313197 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] credential.caching = True {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.313277 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.313437 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] credential.driver = sql {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.313514 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] catalog.caching = True {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.313678 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.313756 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.313908 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.313985 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.314146 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.314225 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.314383 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.314461 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.314615 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.314694 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.314852 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] credential.caching = True {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.314929 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.315101 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] credential.driver = sql {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.315238 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.315371 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.315494 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.315618 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.315698 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.315859 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.315988 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.316112 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.316235 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.316358 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.316482 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.316604 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.316733 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.316858 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.317006 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.317128 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.317257 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] federation.caching = True {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.317257 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.317499 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.317622 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] federation.driver = sql {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.317745 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.317870 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.317992 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.318115 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.318245 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.318368 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.318491 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.318625 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.318744 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.318868 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.318999 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] federation.caching = True {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.319120 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.319250 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.319374 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.319496 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] federation.driver = sql {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.319619 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.319749 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.319877 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.320007 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.320130 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] identity.caching = True {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.320253 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.320389 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.320519 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.320643 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.320764 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.320883 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.321021 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.321140 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.321339 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.321470 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] identity.driver = sql {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.321594 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.321716 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.321838 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.321961 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.322112 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] identity.caching = True {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.322231 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.322354 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.322482 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.322610 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.322733 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.322863 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.322987 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.323108 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.323231 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.323354 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] identity.driver = sql {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.323476 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.323606 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.323729 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.323863 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.323986 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.323986 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.324219 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.324349 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.324472 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.324594 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.324716 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.324837 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.325002 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.325134 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.325260 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.325383 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.325516 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.325639 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.325779 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.325908 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.325986 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.326057 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.326128 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.326200 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.326278 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.326351 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.326422 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.326492 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.326562 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.326631 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.326709 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.326781 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.326851 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.326920 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.326990 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.327061 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.327139 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.327211 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.327282 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.327352 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.327422 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.327491 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.327568 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.327640 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.327750 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.327836 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] ldap.password = **** {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.327982 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.328061 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.328230 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.328330 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.328526 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.328604 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.328761 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.328838 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.329012 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.329094 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.329573 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.329652 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.329805 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.329882 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.330033 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.330112 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.330261 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.330343 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] ldap.password = **** {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.330499 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.330576 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.330738 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.330817 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.330965 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.331041 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.331193 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.331271 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.331427 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.331507 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.331654 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] ldap.user = None {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.331732 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.331890 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.331967 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.332130 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.332207 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.332358 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.332435 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.332600 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.332678 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.332842 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.332921 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.333103 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.333181 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.333332 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.333409 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.333559 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.333642 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.333790 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] ldap.user = None {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.333866 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.334018 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.334095 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.334249 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.334327 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.334475 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.334552 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.334719 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.334797 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.334959 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.335038 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.335199 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.335276 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.335432 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.335508 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.335663 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.335742 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.335888 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.335964 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.336121 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.336203 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.336350 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.336428 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.336579 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.336657 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.336807 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.336883 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.337057 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.337137 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.337297 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.337376 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.337540 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.337618 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.337771 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.337851 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.338000 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.338078 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] policy.driver = sql {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.338235 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.338311 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.338472 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.338609 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.338686 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.338756 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.338920 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] receipt.caching = True {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.338998 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.339152 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.339235 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.339398 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.339511 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.339646 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.339752 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.339884 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.340003 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.340139 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] policy.driver = sql {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.340217 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.340370 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.340455 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] resource.caching = True {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.340614 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.340749 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.340851 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.340930 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] resource.driver = sql {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.341110 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] receipt.caching = True {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.341203 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.341347 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.341425 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.341595 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.341673 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.341827 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] revoke.caching = True {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.341911 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.342058 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.342135 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.342292 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.342368 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.342528 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] role.cache_time = None {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.342613 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] resource.caching = True {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.342759 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] role.caching = True {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.342836 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.343003 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] role.driver = sql {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.343081 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] resource.driver = sql {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.343231 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] role.list_limit = None {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.343383 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.343478 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.343556 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.343721 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.343799 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.343977 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] revoke.caching = True {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.344068 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.344221 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.344299 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.344454 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.344534 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.344693 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] role.cache_time = None {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.344779 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.344928 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] role.caching = True {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.345026 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.345193 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.345320 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] role.driver = sql {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.345445 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.345579 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] role.list_limit = None {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.345704 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.345827 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.345962 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.346084 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.346205 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.346333 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.346457 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.346581 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.346581 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.346802 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.346921 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.347049 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.347170 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.347291 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.347411 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.347541 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.347661 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.347789 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.347909 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.348029 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.348148 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.348270 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.348397 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.348517 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.348636 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.348756 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.348756 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.349002 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.349143 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.349266 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.349386 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.349505 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.349625 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.349744 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.349871 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.349991 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.350111 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.350231 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.350351 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.350469 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.350596 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.350729 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.350849 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.350967 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.351092 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] token.cache_time = None {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.351214 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.351340 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] token.caching = True {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.351461 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.351582 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.351701 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.351821 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] token.provider = fernet {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.351940 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.352067 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.352188 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.352308 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.352437 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.352557 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.352557 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.352788 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.352908 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.353077 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.353198 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.353320 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.353442 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] trust.driver = sql {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.353577 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] token.cache_time = None {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.353701 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.353823 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] token.caching = True {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.353944 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.354078 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.354202 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.354331 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] token.provider = fernet {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.354455 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.354579 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.354701 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.354823 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.354946 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.355080 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.355215 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.355339 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.355462 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.355586 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.355709 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] database.connection = **** {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.355841 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.355974 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.356094 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] trust.driver = sql {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.356094 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.356327 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.356327 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.356561 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.356561 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.356796 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.356796 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.357051 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.357173 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.357293 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.357422 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.357552 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.357685 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.357812 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.357933 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.358052 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.358172 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.358292 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] database.connection = **** {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.358420 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.358540 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.358659 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.358780 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.358900 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.359019 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.359147 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.359283 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.359408 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.359527 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.359696 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.359827 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.359948 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.360067 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.360187 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.360187 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81508) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Aug 28 00:40:28.360413 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.360413 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.360413 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.360413 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.360413 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.360413 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.361109 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.361109 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.361109 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.361458 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.361580 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Aug 28 00:40:28.361727 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81507) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Aug 28 00:40:28.388265 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=81508) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 28 00:40:28.388497 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=81508) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 28 00:40:28.388720 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=81508) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 28 00:40:28.390080 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=81507) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 28 00:40:28.390316 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=81507) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 28 00:40:28.390552 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=81507) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 28 00:40:28.397025 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=81508) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 28 00:40:28.399072 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=81507) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 28 00:40:28.424683 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=81508) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 28 00:40:28.424900 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=81508) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 28 00:40:28.426869 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=81507) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 28 00:40:28.427087 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=81507) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 28 00:40:28.477147 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=81508) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 28 00:40:28.477783 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=81507) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 28 00:40:28.524207 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=81508) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 28 00:40:28.524467 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=81508) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 28 00:40:28.525828 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=81507) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 28 00:40:28.526042 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=81507) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 28 00:40:28.682543 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81508) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 28 00:40:28.682797 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=81508) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 28 00:40:28.682895 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=81508) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 28 00:40:28.683090 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=81508) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 28 00:40:28.683301 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=81508) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 28 00:40:28.683488 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81507) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 28 00:40:28.683583 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=81508) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 28 00:40:28.683728 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=81507) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 28 00:40:28.683933 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=81507) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 28 00:40:28.684154 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=81507) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 28 00:40:28.684371 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=81507) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 28 00:40:28.684588 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=81507) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 28 00:40:28.706604 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81508) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 28 00:40:28.706755 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81507) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 28 00:40:28.706930 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=81508) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 28 00:40:28.707064 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=81507) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 28 00:40:28.707195 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=81508) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 28 00:40:28.707195 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=81508) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 28 00:40:28.707396 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=81507) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 28 00:40:28.707484 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=81508) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 28 00:40:28.707615 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=81507) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 28 00:40:28.707699 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=81508) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 28 00:40:28.707829 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=81507) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 28 00:40:28.707914 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81508) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 28 00:40:28.708036 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=81507) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 28 00:40:28.708036 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81507) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 28 00:40:28.718991 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=81508) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 28 00:40:28.719427 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=81507) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 28 00:40:28.731644 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81508) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 28 00:40:28.731880 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=81508) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 28 00:40:28.732046 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81507) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 28 00:40:28.732134 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=81508) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 28 00:40:28.732197 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=81507) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 28 00:40:28.732265 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=81508) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 28 00:40:28.732439 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=81507) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 28 00:40:28.732568 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=81508) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 28 00:40:28.732722 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=81507) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 28 00:40:28.732806 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=81508) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 28 00:40:28.732870 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=81507) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 28 00:40:28.732937 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81508) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 28 00:40:28.733073 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=81507) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 28 00:40:28.733321 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81507) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Aug 28 00:40:28.749559 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=81508) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.749795 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=81508) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Aug 28 00:40:28.749946 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=81507) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.750129 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=81508) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.750277 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=81507) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Aug 28 00:40:28.750358 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=81508) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Aug 28 00:40:28.750602 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=81507) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.750730 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=81508) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.750807 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=81507) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Aug 28 00:40:28.750871 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=81508) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.751052 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=81507) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.751149 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=81508) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.751338 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=81507) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.751465 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=81508) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.751862 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=81507) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.752126 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=81507) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.760790 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=81508) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.761092 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=81508) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.761359 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=81508) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.761607 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=81507) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.761734 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=81508) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.761941 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=81507) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.762142 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=81507) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.762460 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=81507) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.768540 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=81508) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 28 00:40:28.769488 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=81507) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 28 00:40:28.771421 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=81508) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 28 00:40:28.771703 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=81508) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.771985 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=81508) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.772265 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=81508) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.772393 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=81507) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 28 00:40:28.772612 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=81508) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.772708 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=81507) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.772841 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=81508) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.772988 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=81507) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.773116 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=81508) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.773199 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=81507) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.773342 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=81508) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.773498 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=81507) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.773630 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=81508) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.773759 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=81507) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.773895 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=81508) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.774035 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=81507) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.774216 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=81508) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.774362 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=81507) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.774643 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=81507) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.774922 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=81507) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.775197 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=81507) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.796468 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=81508) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.797722 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=81507) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.798096 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=81508) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 28 00:40:28.798375 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=81508) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.799192 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=81507) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 28 00:40:28.799478 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=81507) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.802853 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=81508) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 28 00:40:28.803129 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=81508) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.803410 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=81508) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.804122 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=81507) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 28 00:40:28.804409 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=81507) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.804692 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=81507) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.809823 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=81508) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 28 00:40:28.810103 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=81508) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.811022 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=81507) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 28 00:40:28.811310 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=81507) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.814059 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=81508) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 28 00:40:28.814347 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=81508) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.814630 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=81508) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.814908 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=81508) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.815153 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=81508) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.815550 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=81507) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 28 00:40:28.815844 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=81507) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.816143 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=81507) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.816432 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=81507) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.816718 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=81507) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.816988 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=81507) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.817140 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=81508) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.817350 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=81507) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.817522 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=81508) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.817658 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=81507) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.817803 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=81508) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.833298 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=81508) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.833554 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=81508) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.835148 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=81507) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.835442 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=81507) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.837464 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=81508) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 28 00:40:28.839334 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=81507) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 28 00:40:28.840496 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=81508) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.840761 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=81508) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.842560 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=81507) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.842866 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=81507) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.845380 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=81508) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 28 00:40:28.847107 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=81507) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 28 00:40:28.848422 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=81508) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 28 00:40:28.850277 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=81507) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 28 00:40:28.851588 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=81508) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.851882 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=81508) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.852160 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=81508) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.852440 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=81508) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.852715 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=81508) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.853126 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=81508) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.854723 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=81507) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.855030 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=81507) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.855318 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=81507) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.855604 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=81507) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.855886 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=81507) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.856175 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=81507) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.868719 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=81508) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 28 00:40:28.869004 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=81508) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.869290 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=81508) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.869558 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=81508) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.871979 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=81507) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 28 00:40:28.872273 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=81507) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.872545 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=81507) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.872813 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=81507) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.876248 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH2: ['/token' {}] {{(pid=81508) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.877922 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=81508) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.879434 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=81508) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.879674 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=81508) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.879848 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH2: ['/token' {}] {{(pid=81507) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.881411 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=81507) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.882337 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=81508) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 28 00:40:28.882624 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=81508) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.882899 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=81508) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.883091 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=81507) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.883181 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=81508) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.883247 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=81507) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.883431 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=81508) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.885865 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=81507) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 28 00:40:28.886158 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=81507) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.886550 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=81507) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.886850 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=81507) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.887134 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=81507) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.894629 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=81508) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 28 00:40:28.894912 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=81508) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.895186 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=81508) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.895465 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=81508) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.895741 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=81508) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.896007 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=81508) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.896277 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=81508) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.898648 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=81507) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 28 00:40:28.898950 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=81507) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.899309 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=81507) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.899593 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=81507) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.899879 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=81507) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.900155 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=81507) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.900434 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=81507) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.911501 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=81508) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 28 00:40:28.914566 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=81508) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 28 00:40:28.915841 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=81507) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 28 00:40:28.917402 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=81508) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.918769 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=81507) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 28 00:40:28.918897 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=81508) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.920267 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=81508) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 28 00:40:28.920545 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=81508) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.920822 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=81508) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.921680 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=81507) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.923172 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=81507) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.924778 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=81507) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 28 00:40:28.925067 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=81507) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.925384 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=81507) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.927283 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=81508) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.928758 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=81508) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 28 00:40:28.931617 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=81508) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.931814 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=81507) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.931907 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=81508) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.932171 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=81508) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.932461 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=81508) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.933327 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=81507) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 28 00:40:28.936255 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=81507) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.936548 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=81507) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.936835 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=81507) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.937160 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=81507) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.940312 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=81508) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 28 00:40:28.940601 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=81508) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.940904 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=81508) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.944911 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=81507) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 28 00:40:28.945235 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=81507) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.945522 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=81507) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.947687 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=81508) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 28 00:40:28.947976 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=81508) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.948243 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=81508) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.948509 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=81508) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.948774 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=81508) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.949066 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=81508) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.949357 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=81508) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.949621 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=81508) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.949885 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=81508) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.950150 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=81508) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.950397 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=81508) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.950652 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=81508) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.950935 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=81508) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.951198 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=81508) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.952349 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=81507) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Aug 28 00:40:28.952638 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=81507) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.952920 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=81507) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.953400 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=81507) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.953676 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=81507) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.953953 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=81507) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.954596 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=81507) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.954876 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=81507) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.955155 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=81507) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.955426 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=81507) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.955692 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=81507) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.955952 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=81507) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.956237 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=81507) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.956508 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=81507) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Aug 28 00:40:28.986802 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: WSGI app 0 (mountpoint='') ready in 5 seconds on interpreter 0x56283b1cf9c0 pid: 81508 (default app) Aug 28 00:40:28.990307 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64a7f439-5dab-4fde-87bc-c002747adf95 None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:40:28.990434 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: WSGI app 0 (mountpoint='') ready in 5 seconds on interpreter 0x56283b1cf9c0 pid: 81507 (default app) Aug 28 00:40:28.990588 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64a7f439-5dab-4fde-87bc-c002747adf95 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:40:28.990856 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64a7f439-5dab-4fde-87bc-c002747adf95 None None] PATH_INFO: `/v3` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:40:28.995000 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1/1] 10.222.0.42 () {50 vars in 791 bytes} [Sun Aug 28 00:40:28 2022] GET /identity/v3/ => generated 255 bytes in 8 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:40:35.568516 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a47ef9-8619-4f57-ae60-5c3b110e82e6 None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:40:35.569112 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a47ef9-8619-4f57-ae60-5c3b110e82e6 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:40:35.569112 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a47ef9-8619-4f57-ae60-5c3b110e82e6 None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:40:35.573371 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1/2] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:40:35 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:40:35.579013 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ae8703e-2a78-4d2c-83cd-776da5f713ab None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:40:35.579851 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ae8703e-2a78-4d2c-83cd-776da5f713ab None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:40:35.579851 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ae8703e-2a78-4d2c-83cd-776da5f713ab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:40:35.649149 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: INFO dbcounter [None req-0ae8703e-2a78-4d2c-83cd-776da5f713ab None None] Registered counter for database keystone Aug 28 00:40:35.668719 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG oslo_db.sqlalchemy.engines [None req-0ae8703e-2a78-4d2c-83cd-776da5f713ab None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=81508) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Aug 28 00:40:35.676276 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writer thread running {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Aug 28 00:40:35.860213 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG passlib.handlers.bcrypt [None req-0ae8703e-2a78-4d2c-83cd-776da5f713ab None None] detected 'bcrypt' backend, version '3.2.2' {{(pid=81508) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Aug 28 00:40:35.860525 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG passlib.handlers.bcrypt [None req-0ae8703e-2a78-4d2c-83cd-776da5f713ab None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=81508) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Aug 28 00:40:35.915273 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-0ae8703e-2a78-4d2c-83cd-776da5f713ab None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:40:35.916893 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-0ae8703e-2a78-4d2c-83cd-776da5f713ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:40:36.049358 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 2/3] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:40:35 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 471 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Aug 28 00:40:36.369414 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85a47ef9-8619-4f57-ae60-5c3b110e82e6 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:40:36.439612 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: INFO dbcounter [None req-85a47ef9-8619-4f57-ae60-5c3b110e82e6 None None] Registered counter for database keystone Aug 28 00:40:36.461538 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG oslo_db.sqlalchemy.engines [None req-85a47ef9-8619-4f57-ae60-5c3b110e82e6 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=81507) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Aug 28 00:40:36.469429 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writer thread running {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Aug 28 00:40:36.589151 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85a47ef9-8619-4f57-ae60-5c3b110e82e6 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:40:36.594412 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a480b34c-e0d3-4a5c-ab9d-6a8d81538998 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:40:36.595056 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a480b34c-e0d3-4a5c-ab9d-6a8d81538998 admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:40:36.595333 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a480b34c-e0d3-4a5c-ab9d-6a8d81538998 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:40:36.595602 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a480b34c-e0d3-4a5c-ab9d-6a8d81538998 admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:40:36.604806 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a480b34c-e0d3-4a5c-ab9d-6a8d81538998 admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:40:36.655684 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: WARNING keystone.common.rbac_enforcer.enforcer [None req-a480b34c-e0d3-4a5c-ab9d-6a8d81538998 admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Aug 28 00:40:36.656021 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a480b34c-e0d3-4a5c-ab9d-6a8d81538998 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:40:36.666642 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 2/4] 10.222.0.42 () {58 vars in 1324 bytes} [Sun Aug 28 00:40:36 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 298 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 28 00:40:36.672913 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ae8703e-2a78-4d2c-83cd-776da5f713ab None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:40:36.687554 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ae8703e-2a78-4d2c-83cd-776da5f713ab None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:40:36.692688 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc99c01e-4aac-4b1c-bdab-c2989ec07778 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:40:36.693874 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc99c01e-4aac-4b1c-bdab-c2989ec07778 admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:40:36.694157 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc99c01e-4aac-4b1c-bdab-c2989ec07778 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:40:36.694430 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc99c01e-4aac-4b1c-bdab-c2989ec07778 admin admin] PATH_INFO: `/v3/projects/cca4b1c3ff844aae827dd439e67820c7` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:40:36.696117 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc99c01e-4aac-4b1c-bdab-c2989ec07778 admin admin] RBAC: Authorizing `identity:get_project(project_id=cca4b1c3ff844aae827dd439e67820c7)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:40:36.748711 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: WARNING keystone.common.rbac_enforcer.enforcer [None req-cc99c01e-4aac-4b1c-bdab-c2989ec07778 admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Aug 28 00:40:36.749080 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc99c01e-4aac-4b1c-bdab-c2989ec07778 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:40:36.753866 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 3/5] 10.222.0.42 () {56 vars in 1207 bytes} [Sun Aug 28 00:40:36 2022] GET /identity/v3/projects/cca4b1c3ff844aae827dd439e67820c7 => generated 344 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:40:38.633470 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a26b9d5-0b70-4f47-831d-733fa3d6ffde None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:40:38.634006 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a26b9d5-0b70-4f47-831d-733fa3d6ffde None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:40:38.634006 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a26b9d5-0b70-4f47-831d-733fa3d6ffde None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:40:38.638073 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 3/6] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:40:38 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:40:38.643986 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b475e0-2f90-406b-82e7-41298756e818 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:40:38.644386 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b475e0-2f90-406b-82e7-41298756e818 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:40:38.644588 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b475e0-2f90-406b-82e7-41298756e818 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:40:38.666427 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-c9b475e0-2f90-406b-82e7-41298756e818 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:40:38.667982 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-c9b475e0-2f90-406b-82e7-41298756e818 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:40:38.678048 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 4/7] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:40:38 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Aug 28 00:40:38.938474 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a26b9d5-0b70-4f47-831d-733fa3d6ffde None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:40:38.951337 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a26b9d5-0b70-4f47-831d-733fa3d6ffde None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:40:38.955989 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-909aeed8-0a69-4e7d-ba62-acaa318c2552 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:40:38.956592 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-909aeed8-0a69-4e7d-ba62-acaa318c2552 admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:40:38.956869 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-909aeed8-0a69-4e7d-ba62-acaa318c2552 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:40:38.957164 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-909aeed8-0a69-4e7d-ba62-acaa318c2552 admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:40:38.965381 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-909aeed8-0a69-4e7d-ba62-acaa318c2552 admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:40:38.967201 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-909aeed8-0a69-4e7d-ba62-acaa318c2552 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:40:38.976476 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 4/8] 10.222.0.42 () {58 vars in 1324 bytes} [Sun Aug 28 00:40:38 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 28 00:40:38.982263 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9b475e0-2f90-406b-82e7-41298756e818 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:40:38.992142 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9b475e0-2f90-406b-82e7-41298756e818 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:40:38.997939 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1f16ed6-3736-4e94-ae6f-897ab26503fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:40:38.998588 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f16ed6-3736-4e94-ae6f-897ab26503fc admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:40:38.998870 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f16ed6-3736-4e94-ae6f-897ab26503fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:40:38.999140 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f16ed6-3736-4e94-ae6f-897ab26503fc admin admin] PATH_INFO: `/v3/users/e5cb815a43fc4460b6ddf0175f087128` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:40:39.000682 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1f16ed6-3736-4e94-ae6f-897ab26503fc admin admin] RBAC: Authorizing `identity:get_user(user_id=e5cb815a43fc4460b6ddf0175f087128)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:40:39.002606 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1f16ed6-3736-4e94-ae6f-897ab26503fc admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:40:39.004156 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 5/9] 10.222.0.42 () {56 vars in 1198 bytes} [Sun Aug 28 00:40:38 2022] GET /identity/v3/users/e5cb815a43fc4460b6ddf0175f087128 => generated 247 bytes in 22 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:40:40.960805 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-697b07bb-1d81-469a-8258-9de20a4decf3 None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:40:40.961397 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-697b07bb-1d81-469a-8258-9de20a4decf3 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:40:40.961397 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-697b07bb-1d81-469a-8258-9de20a4decf3 None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:40:40.962665 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 5/10] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:40:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:40:40.968200 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-986d887c-4195-4338-8abb-01d89f101b02 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:40:40.968625 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-986d887c-4195-4338-8abb-01d89f101b02 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:40:40.968720 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-986d887c-4195-4338-8abb-01d89f101b02 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:40:40.987221 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-986d887c-4195-4338-8abb-01d89f101b02 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:40:40.988406 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-986d887c-4195-4338-8abb-01d89f101b02 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:40:41.026651 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dfa584a-903b-489b-b336-e239477b9553 None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:40:41.026935 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dfa584a-903b-489b-b336-e239477b9553 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:40:41.027198 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dfa584a-903b-489b-b336-e239477b9553 None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:40:41.028497 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 6/11] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:40:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:40:41.034085 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88bf932e-25ce-4db7-a199-f9a9d1b34f5a None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:40:41.034366 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88bf932e-25ce-4db7-a199-f9a9d1b34f5a None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:40:41.034636 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88bf932e-25ce-4db7-a199-f9a9d1b34f5a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:40:41.070798 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 6/12] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:40:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 28 00:40:41.076841 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG passlib.handlers.bcrypt [None req-88bf932e-25ce-4db7-a199-f9a9d1b34f5a None None] detected 'bcrypt' backend, version '3.2.2' {{(pid=81507) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Aug 28 00:40:41.077296 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG passlib.handlers.bcrypt [None req-88bf932e-25ce-4db7-a199-f9a9d1b34f5a None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=81507) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Aug 28 00:40:41.122899 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-88bf932e-25ce-4db7-a199-f9a9d1b34f5a None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:40:41.124496 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-88bf932e-25ce-4db7-a199-f9a9d1b34f5a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:40:41.227171 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 7/13] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:40:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 28 00:40:41.335887 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-986d887c-4195-4338-8abb-01d89f101b02 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:40:41.360396 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:40:41.440496 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-986d887c-4195-4338-8abb-01d89f101b02 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:40:41.475357 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-049dc05a-ba45-41fe-907a-a213c8b4f45f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:40:41.475979 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-049dc05a-ba45-41fe-907a-a213c8b4f45f None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:40:41.476258 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-049dc05a-ba45-41fe-907a-a213c8b4f45f None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:40:41.476528 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-049dc05a-ba45-41fe-907a-a213c8b4f45f None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:40:41.488087 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-049dc05a-ba45-41fe-907a-a213c8b4f45f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 28 00:40:41.492026 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88bf932e-25ce-4db7-a199-f9a9d1b34f5a None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:40:41.496899 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-049dc05a-ba45-41fe-907a-a213c8b4f45f None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 28 00:40:41.496899 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:40:41.496899 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:40:41.496899 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:40:41.496899 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:40:41.496899 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:40:41.496899 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:40:41.496899 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:40:41.496899 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:40:41.496899 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:40:41.496899 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:40:41.496899 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:40:41.496899 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 28 00:40:41.496899 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 28 00:40:41.496899 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 28 00:40:41.496899 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 28 00:40:41.498075 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 28 00:40:41.498075 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 28 00:40:41.498075 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 28 00:40:41.498075 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 28 00:40:41.498075 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:40:41.498075 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:40:41.498075 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:40:41.498075 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:40:41.498075 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:40:41.498075 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:40:41.498075 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:40:41.498075 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:40:41.498075 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:40:41.498075 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:40:41.498075 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:40:41.498075 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:40:41.499207 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:40:41.499207 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:40:41.499207 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:40:41.499207 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:40:41.499207 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 28 00:40:41.499207 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 28 00:40:41.499207 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 28 00:40:41.499207 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 28 00:40:41.499207 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 28 00:40:41.499207 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 28 00:40:41.499207 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 28 00:40:41.499207 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:40:41.499207 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 7/14] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Aug 28 00:40:41 2022] GET /identity/v3/roles/admin => generated 83 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:40:41.505986 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-049dc05a-ba45-41fe-907a-a213c8b4f45f None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:40:41.590566 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:40:41.594751 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88bf932e-25ce-4db7-a199-f9a9d1b34f5a None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:40:41.598247 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:40:41.610194 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-049dc05a-ba45-41fe-907a-a213c8b4f45f None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:40:41.634702 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fc65451-2779-4990-afe8-aafb4247148b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:40:41.635593 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fc65451-2779-4990-afe8-aafb4247148b None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:40:41.635877 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fc65451-2779-4990-afe8-aafb4247148b None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:40:41.636149 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fc65451-2779-4990-afe8-aafb4247148b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:40:41.644392 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f23718d0-f7f5-45cf-ade6-de8a302b614d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:40:41.645006 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f23718d0-f7f5-45cf-ade6-de8a302b614d None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:40:41.645319 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f23718d0-f7f5-45cf-ade6-de8a302b614d None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:40:41.645587 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f23718d0-f7f5-45cf-ade6-de8a302b614d None admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:40:41.646270 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f23718d0-f7f5-45cf-ade6-de8a302b614d None admin] RBAC: Adding query filter params (name=admin) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:40:41.646572 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f23718d0-f7f5-45cf-ade6-de8a302b614d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:40:41.647997 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f23718d0-f7f5-45cf-ade6-de8a302b614d None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:40:41.659981 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 8/15] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Aug 28 00:40:41 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:40:41.665571 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f23718d0-f7f5-45cf-ade6-de8a302b614d None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:40:41.704202 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fc65451-2779-4990-afe8-aafb4247148b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:40:41.706103 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fc65451-2779-4990-afe8-aafb4247148b None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:40:41.767839 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f23718d0-f7f5-45cf-ade6-de8a302b614d None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:40:41.803710 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-123501b6-5480-4d0e-ae90-bd4a7bcb6008 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:40:41.804361 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-123501b6-5480-4d0e-ae90-bd4a7bcb6008 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:40:41.804649 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-123501b6-5480-4d0e-ae90-bd4a7bcb6008 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:40:41.804905 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-123501b6-5480-4d0e-ae90-bd4a7bcb6008 None admin] PATH_INFO: `/v3/users/e5cb815a43fc4460b6ddf0175f087128` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:40:41.806156 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-123501b6-5480-4d0e-ae90-bd4a7bcb6008 None admin] RBAC: Authorizing `identity:get_user(user_id=e5cb815a43fc4460b6ddf0175f087128)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:40:41.806321 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 8/16] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Aug 28 00:40:41 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 315 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 28 00:40:41.807673 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-123501b6-5480-4d0e-ae90-bd4a7bcb6008 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:40:41.809368 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 9/17] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Aug 28 00:40:41 2022] GET /identity/v3/users/e5cb815a43fc4460b6ddf0175f087128 => generated 247 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 28 00:40:41.812329 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fc65451-2779-4990-afe8-aafb4247148b None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:40:41.814994 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-123501b6-5480-4d0e-ae90-bd4a7bcb6008 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:40:41.815792 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:40:41.831297 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:40:41.914193 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fc65451-2779-4990-afe8-aafb4247148b None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:40:41.919257 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-123501b6-5480-4d0e-ae90-bd4a7bcb6008 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:40:41.949201 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a9dcdca-9b6c-4163-a501-6b5b098e4ebd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:40:41.950066 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a9dcdca-9b6c-4163-a501-6b5b098e4ebd None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:40:41.950344 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a9dcdca-9b6c-4163-a501-6b5b098e4ebd None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:40:41.950606 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a9dcdca-9b6c-4163-a501-6b5b098e4ebd None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:40:41.955752 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0c03dbf-4a3e-4c89-b1e0-196fa0e82ff4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:40:41.956416 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c03dbf-4a3e-4c89-b1e0-196fa0e82ff4 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:40:41.956688 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c03dbf-4a3e-4c89-b1e0-196fa0e82ff4 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:40:41.956981 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c03dbf-4a3e-4c89-b1e0-196fa0e82ff4 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:40:41.958201 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0c03dbf-4a3e-4c89-b1e0-196fa0e82ff4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:40:41.959651 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0c03dbf-4a3e-4c89-b1e0-196fa0e82ff4 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:40:41.961197 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 10/18] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:40:41 2022] GET /identity/v3/domains/default => generated 199 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:40:41.967105 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0c03dbf-4a3e-4c89-b1e0-196fa0e82ff4 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:40:41.970058 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a9dcdca-9b6c-4163-a501-6b5b098e4ebd None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:40:41.971831 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a9dcdca-9b6c-4163-a501-6b5b098e4ebd None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:40:42.000236 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-3a9dcdca-9b6c-4163-a501-6b5b098e4ebd None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:40:42.000236 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:40:42.000236 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:40:42.000236 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:40:42.000236 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:40:42.000236 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:40:42.000236 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:40:42.000236 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:40:42.000236 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:40:42.000236 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:40:42.000236 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:40:42.000236 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:40:42.000236 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 28 00:40:42.000236 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 28 00:40:42.000236 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 28 00:40:42.000236 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 28 00:40:42.001248 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:40:42.001248 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:40:42.001248 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:40:42.001248 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:40:42.001248 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:40:42.001248 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:40:42.001248 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:40:42.001248 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:40:42.001248 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:40:42.001248 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:40:42.001248 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:40:42.001248 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:40:42.001248 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:40:42.001248 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:40:42.001248 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:40:42.001248 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:40:42.001248 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 28 00:40:42.002076 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 28 00:40:42.002076 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:40:42.002076 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:40:42.002590 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 9/19] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:40:41 2022] GET /identity/v3/domains/Default => generated 87 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:40:42.008447 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a9dcdca-9b6c-4163-a501-6b5b098e4ebd None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:40:42.050976 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:40:42.070226 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:40:42.071610 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0c03dbf-4a3e-4c89-b1e0-196fa0e82ff4 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:40:42.107645 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d06d77b-8206-4ca1-a9d1-9b43225a62de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:40:42.108278 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d06d77b-8206-4ca1-a9d1-9b43225a62de None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:40:42.108551 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d06d77b-8206-4ca1-a9d1-9b43225a62de None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:40:42.108817 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d06d77b-8206-4ca1-a9d1-9b43225a62de None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:40:42.109599 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d06d77b-8206-4ca1-a9d1-9b43225a62de None admin] RBAC: Adding query filter params (role.id=f08d6043261747e3b4763c744eef7dfb, scope.domain.id=default, user.id=e5cb815a43fc4460b6ddf0175f087128) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:40:42.109897 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d06d77b-8206-4ca1-a9d1-9b43225a62de None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:40:42.111357 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d06d77b-8206-4ca1-a9d1-9b43225a62de None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:40:42.117312 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a9dcdca-9b6c-4163-a501-6b5b098e4ebd None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:40:42.120172 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 11/20] 10.222.0.42 () {56 vars in 1322 bytes} [Sun Aug 28 00:40:41 2022] GET /identity/v3/role_assignments?user.id=e5cb815a43fc4460b6ddf0175f087128&scope.domain.id=default&role.id=f08d6043261747e3b4763c744eef7dfb => generated 233 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:40:42.153678 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fa87429-ebf8-49ca-b966-05e8098b274e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:40:42.154308 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa87429-ebf8-49ca-b966-05e8098b274e None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:40:42.154590 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa87429-ebf8-49ca-b966-05e8098b274e None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:40:42.154857 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa87429-ebf8-49ca-b966-05e8098b274e None admin] PATH_INFO: `/v3/domains` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:40:42.155514 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fa87429-ebf8-49ca-b966-05e8098b274e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:40:42.155809 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fa87429-ebf8-49ca-b966-05e8098b274e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:40:42.157363 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fa87429-ebf8-49ca-b966-05e8098b274e None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:40:42.177551 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 10/21] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Aug 28 00:40:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:40:44.377222 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2efdc9bb-90e8-442c-8ff6-2f717ef36305 None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:40:44.377712 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2efdc9bb-90e8-442c-8ff6-2f717ef36305 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:40:44.377786 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2efdc9bb-90e8-442c-8ff6-2f717ef36305 None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:40:44.379383 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 12/22] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:40:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:40:44.383480 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa1bce5-7458-49ca-ad22-d9a34653c95e None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:40:44.384300 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa1bce5-7458-49ca-ad22-d9a34653c95e None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:40:44.384410 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca54509e-4af5-4c2c-85c9-4e70dcba5b88 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:40:44.384676 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca54509e-4af5-4c2c-85c9-4e70dcba5b88 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:40:44.384971 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca54509e-4af5-4c2c-85c9-4e70dcba5b88 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:40:44.385225 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa1bce5-7458-49ca-ad22-d9a34653c95e None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:40:44.387019 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 11/23] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:40:44 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:40:44.410045 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-ca54509e-4af5-4c2c-85c9-4e70dcba5b88 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:40:44.411276 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-ca54509e-4af5-4c2c-85c9-4e70dcba5b88 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:40:44.421923 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18cde9d7-2de8-47da-8cd3-976940b35c21 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:40:44.422493 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18cde9d7-2de8-47da-8cd3-976940b35c21 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:40:44.422958 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18cde9d7-2de8-47da-8cd3-976940b35c21 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:40:44.451567 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-18cde9d7-2de8-47da-8cd3-976940b35c21 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:40:44.453914 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-18cde9d7-2de8-47da-8cd3-976940b35c21 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:40:44.520987 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 13/24] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:40:44 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 28 00:40:44.522117 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c081a9f-cf4b-4579-b020-60d6c0b3065f None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:40:44.522410 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c081a9f-cf4b-4579-b020-60d6c0b3065f None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:40:44.522683 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c081a9f-cf4b-4579-b020-60d6c0b3065f None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:40:44.524228 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 14/25] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:40:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:40:44.531414 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2995d019-c052-422d-9b72-254627ae8cdb None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:40:44.531704 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2995d019-c052-422d-9b72-254627ae8cdb None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:40:44.531983 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2995d019-c052-422d-9b72-254627ae8cdb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:40:44.541533 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 12/26] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:40:44 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 28 00:40:44.570177 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-2995d019-c052-422d-9b72-254627ae8cdb None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:40:44.571407 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-2995d019-c052-422d-9b72-254627ae8cdb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:40:44.664178 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 15/27] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:40:44 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 28 00:40:45.007305 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18cde9d7-2de8-47da-8cd3-976940b35c21 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:40:45.027648 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c1d374d-9405-4817-8e03-a5b7f461cb39 None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:40:45.028050 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c1d374d-9405-4817-8e03-a5b7f461cb39 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:40:45.028224 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c1d374d-9405-4817-8e03-a5b7f461cb39 None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:40:45.030323 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 16/28] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:40:45 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 28 00:40:45.037070 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9be9a9a-56a1-4e77-a1d4-22580250a7af None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:40:45.037403 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9be9a9a-56a1-4e77-a1d4-22580250a7af None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:40:45.037692 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9be9a9a-56a1-4e77-a1d4-22580250a7af None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:40:45.058727 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:40:45.087706 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-d9be9a9a-56a1-4e77-a1d4-22580250a7af None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:40:45.089463 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-d9be9a9a-56a1-4e77-a1d4-22580250a7af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:40:45.102438 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:40:45.164399 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18cde9d7-2de8-47da-8cd3-976940b35c21 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:40:45.202469 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d69cf9b-5645-486a-bcea-8efb9e272a76 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:40:45.203711 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d69cf9b-5645-486a-bcea-8efb9e272a76 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:40:45.204191 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d69cf9b-5645-486a-bcea-8efb9e272a76 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:40:45.205044 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d69cf9b-5645-486a-bcea-8efb9e272a76 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:40:45.216840 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d69cf9b-5645-486a-bcea-8efb9e272a76 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:40:45.218641 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d69cf9b-5645-486a-bcea-8efb9e272a76 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:40:45.221500 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 17/29] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:40:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 28 00:40:45.222854 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9be9a9a-56a1-4e77-a1d4-22580250a7af None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:40:45.236058 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-7d69cf9b-5645-486a-bcea-8efb9e272a76 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:40:45.236058 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:40:45.236058 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:40:45.236058 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:40:45.236058 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:40:45.236058 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:40:45.236058 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:40:45.236058 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:40:45.236058 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:40:45.236058 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:40:45.236058 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:40:45.236058 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:40:45.236058 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 28 00:40:45.236058 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 28 00:40:45.236058 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 28 00:40:45.236058 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 28 00:40:45.236949 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:40:45.236949 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:40:45.236949 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:40:45.236949 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:40:45.236949 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:40:45.236949 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:40:45.236949 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:40:45.236949 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:40:45.236949 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:40:45.236949 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:40:45.236949 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:40:45.236949 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:40:45.236949 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:40:45.236949 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:40:45.236949 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:40:45.236949 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:40:45.236949 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 28 00:40:45.237794 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 28 00:40:45.237794 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:40:45.237794 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:40:45.240050 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 13/30] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:40:45 2022] GET /identity/v3/domains/Default => generated 87 bytes in 233 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:40:45.241064 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d69cf9b-5645-486a-bcea-8efb9e272a76 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:40:45.372109 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9be9a9a-56a1-4e77-a1d4-22580250a7af None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:40:45.405917 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:40:45.410208 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e0abf2b-63b4-4f74-80ff-16dee63b4dc2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:40:45.410899 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e0abf2b-63b4-4f74-80ff-16dee63b4dc2 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:40:45.411225 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e0abf2b-63b4-4f74-80ff-16dee63b4dc2 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:40:45.411565 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e0abf2b-63b4-4f74-80ff-16dee63b4dc2 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:40:45.413557 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:40:45.416975 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d69cf9b-5645-486a-bcea-8efb9e272a76 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:40:45.437626 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e0abf2b-63b4-4f74-80ff-16dee63b4dc2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 28 00:40:45.443078 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-8e0abf2b-63b4-4f74-80ff-16dee63b4dc2 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 28 00:40:45.443078 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:40:45.443078 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:40:45.443078 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:40:45.443078 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:40:45.443078 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:40:45.443078 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:40:45.443078 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:40:45.443078 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:40:45.443078 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:40:45.443078 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:40:45.443078 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:40:45.443078 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 28 00:40:45.443078 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 28 00:40:45.443078 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 28 00:40:45.443078 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 28 00:40:45.444010 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 28 00:40:45.444010 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 28 00:40:45.444010 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 28 00:40:45.444010 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 28 00:40:45.444010 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:40:45.444010 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:40:45.444010 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:40:45.444010 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:40:45.444010 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:40:45.444010 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:40:45.444010 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:40:45.444010 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:40:45.444010 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:40:45.444010 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:40:45.444010 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:40:45.444010 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:40:45.444773 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:40:45.444773 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:40:45.444773 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:40:45.444773 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:40:45.444773 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 28 00:40:45.444773 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 28 00:40:45.444773 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 28 00:40:45.444773 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 28 00:40:45.444773 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 28 00:40:45.444773 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 28 00:40:45.444773 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 28 00:40:45.444773 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:40:45.456037 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 18/31] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Aug 28 00:40:45 2022] GET /identity/v3/roles/admin => generated 83 bytes in 228 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:40:45.456994 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e0abf2b-63b4-4f74-80ff-16dee63b4dc2 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:40:45.463286 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c49782e0-3163-4f50-8e72-a5ba6114600a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:40:45.463926 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c49782e0-3163-4f50-8e72-a5ba6114600a None admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:40:45.464203 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c49782e0-3163-4f50-8e72-a5ba6114600a None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:40:45.464472 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c49782e0-3163-4f50-8e72-a5ba6114600a None admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:40:45.465193 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c49782e0-3163-4f50-8e72-a5ba6114600a None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:40:45.466622 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c49782e0-3163-4f50-8e72-a5ba6114600a None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:40:45.594858 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e0abf2b-63b4-4f74-80ff-16dee63b4dc2 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:40:45.634298 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d05ed0f8-b944-4b45-8824-3be5743c3076 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:40:45.634913 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d05ed0f8-b944-4b45-8824-3be5743c3076 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:40:45.635192 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d05ed0f8-b944-4b45-8824-3be5743c3076 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:40:45.635463 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d05ed0f8-b944-4b45-8824-3be5743c3076 None admin] PATH_INFO: `/v3/domains` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:40:45.636161 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d05ed0f8-b944-4b45-8824-3be5743c3076 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:40:45.636462 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d05ed0f8-b944-4b45-8824-3be5743c3076 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:40:45.637941 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d05ed0f8-b944-4b45-8824-3be5743c3076 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:40:45.638607 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 14/32] 10.222.0.42 () {60 vars in 1131 bytes} [Sun Aug 28 00:40:45 2022] POST /identity/v3/roles => generated 219 bytes in 398 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:40:45.639620 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c49782e0-3163-4f50-8e72-a5ba6114600a None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:40:45.658220 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 19/33] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Aug 28 00:40:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:40:45.659168 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d05ed0f8-b944-4b45-8824-3be5743c3076 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:40:45.761400 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c49782e0-3163-4f50-8e72-a5ba6114600a None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:40:45.800154 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6fb7898-6d81-412e-86f5-c4f5916287b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:40:45.800812 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6fb7898-6d81-412e-86f5-c4f5916287b9 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:40:45.801147 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6fb7898-6d81-412e-86f5-c4f5916287b9 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:40:45.801428 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6fb7898-6d81-412e-86f5-c4f5916287b9 None admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:40:45.802126 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6fb7898-6d81-412e-86f5-c4f5916287b9 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:40:45.802437 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6fb7898-6d81-412e-86f5-c4f5916287b9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:40:45.803476 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:40:45.805872 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6fb7898-6d81-412e-86f5-c4f5916287b9 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:40:45.825629 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 15/34] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Aug 28 00:40:45 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:40:45.826510 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6fb7898-6d81-412e-86f5-c4f5916287b9 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:40:45.853767 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:40:45.876192 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d05ed0f8-b944-4b45-8824-3be5743c3076 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:40:45.921624 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c285cb7-f90b-4c05-8c70-ce21187159e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:40:45.922271 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c285cb7-f90b-4c05-8c70-ce21187159e8 None admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:40:45.922551 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c285cb7-f90b-4c05-8c70-ce21187159e8 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:40:45.922825 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c285cb7-f90b-4c05-8c70-ce21187159e8 None admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:40:45.923676 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c285cb7-f90b-4c05-8c70-ce21187159e8 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:40:45.925098 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c285cb7-f90b-4c05-8c70-ce21187159e8 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:40:45.955613 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6fb7898-6d81-412e-86f5-c4f5916287b9 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:40:45.997930 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82204c85-7605-4255-8651-693a96d8afaf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:40:45.998559 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82204c85-7605-4255-8651-693a96d8afaf None admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:40:45.998848 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82204c85-7605-4255-8651-693a96d8afaf None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:40:45.999129 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82204c85-7605-4255-8651-693a96d8afaf None admin] PATH_INFO: `/v3/projects` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:40:46.000086 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82204c85-7605-4255-8651-693a96d8afaf None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:40:46.001602 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82204c85-7605-4255-8651-693a96d8afaf None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:40:46.034147 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 16/35] 10.222.0.42 () {60 vars in 1141 bytes} [Sun Aug 28 00:40:45 2022] POST /identity/v3/projects => generated 301 bytes in 208 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:40:46.035437 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82204c85-7605-4255-8651-693a96d8afaf None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:40:46.058111 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 20/36] 10.222.0.42 () {60 vars in 1131 bytes} [Sun Aug 28 00:40:45 2022] POST /identity/v3/roles => generated 225 bytes in 394 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:40:46.058111 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2ee3de7-2fab-4970-b29e-a8820efce666 None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:40:46.058111 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2ee3de7-2fab-4970-b29e-a8820efce666 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:40:46.058111 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2ee3de7-2fab-4970-b29e-a8820efce666 None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:40:46.058111 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 21/37] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:40:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:40:46.062871 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db20b982-16c1-4356-a45b-01c95c03a9cd None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:40:46.063184 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db20b982-16c1-4356-a45b-01c95c03a9cd None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:40:46.063474 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db20b982-16c1-4356-a45b-01c95c03a9cd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:40:46.084600 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-db20b982-16c1-4356-a45b-01c95c03a9cd None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:40:46.086334 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-db20b982-16c1-4356-a45b-01c95c03a9cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:40:46.203161 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:40:46.242992 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 22/38] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:40:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 28 00:40:46.258735 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5706e0c-1b14-422b-b106-737da81bf9b5 None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:40:46.259087 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5706e0c-1b14-422b-b106-737da81bf9b5 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:40:46.259376 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5706e0c-1b14-422b-b106-737da81bf9b5 None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:40:46.260829 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 23/39] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:40:46 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 28 00:40:46.261094 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82204c85-7605-4255-8651-693a96d8afaf None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:40:46.268166 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef2c54e5-95dc-4482-9920-b64c4609c08d None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:40:46.268455 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef2c54e5-95dc-4482-9920-b64c4609c08d None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:40:46.268734 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef2c54e5-95dc-4482-9920-b64c4609c08d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:40:46.293658 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-ef2c54e5-95dc-4482-9920-b64c4609c08d None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:40:46.294985 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-ef2c54e5-95dc-4482-9920-b64c4609c08d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:40:46.304792 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da5fd862-4a2d-410d-866d-d0269bfd375f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:40:46.306485 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da5fd862-4a2d-410d-866d-d0269bfd375f None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:40:46.307038 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da5fd862-4a2d-410d-866d-d0269bfd375f None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:40:46.307540 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da5fd862-4a2d-410d-866d-d0269bfd375f None admin] PATH_INFO: `/v3/users/e5cb815a43fc4460b6ddf0175f087128` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:40:46.309904 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da5fd862-4a2d-410d-866d-d0269bfd375f None admin] RBAC: Authorizing `identity:get_user(user_id=e5cb815a43fc4460b6ddf0175f087128)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:40:46.311872 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da5fd862-4a2d-410d-866d-d0269bfd375f None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:40:46.314472 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 17/40] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Aug 28 00:40:46 2022] GET /identity/v3/users/e5cb815a43fc4460b6ddf0175f087128 => generated 247 bytes in 279 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:40:46.320121 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da5fd862-4a2d-410d-866d-d0269bfd375f None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:40:46.355295 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:40:46.389842 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 24/41] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:40:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 28 00:40:46.446995 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da5fd862-4a2d-410d-866d-d0269bfd375f None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:40:46.482996 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6072eb1d-614a-43fb-b18b-ff55cec9f9d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:40:46.483615 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6072eb1d-614a-43fb-b18b-ff55cec9f9d7 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:40:46.483899 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6072eb1d-614a-43fb-b18b-ff55cec9f9d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:40:46.484177 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6072eb1d-614a-43fb-b18b-ff55cec9f9d7 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:40:46.485357 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6072eb1d-614a-43fb-b18b-ff55cec9f9d7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:40:46.486832 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6072eb1d-614a-43fb-b18b-ff55cec9f9d7 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:40:46.488593 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 18/42] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:40:46 2022] GET /identity/v3/domains/default => generated 199 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 28 00:40:46.494211 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef2c54e5-95dc-4482-9920-b64c4609c08d None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:40:46.532255 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6072eb1d-614a-43fb-b18b-ff55cec9f9d7 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:40:46.545164 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:40:46.648256 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6072eb1d-614a-43fb-b18b-ff55cec9f9d7 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:40:46.683334 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de30f32d-b418-4288-8829-0aacfa613466 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:40:46.683959 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de30f32d-b418-4288-8829-0aacfa613466 None admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:40:46.684246 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de30f32d-b418-4288-8829-0aacfa613466 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:40:46.684520 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de30f32d-b418-4288-8829-0aacfa613466 None admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:40:46.685254 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de30f32d-b418-4288-8829-0aacfa613466 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:40:46.686666 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de30f32d-b418-4288-8829-0aacfa613466 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:40:46.702567 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 19/43] 10.222.0.42 () {60 vars in 1131 bytes} [Sun Aug 28 00:40:46 2022] POST /identity/v3/roles => generated 223 bytes in 171 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 28 00:40:46.703451 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de30f32d-b418-4288-8829-0aacfa613466 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:40:46.707334 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef2c54e5-95dc-4482-9920-b64c4609c08d None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:40:46.749061 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2f671b9-bb4c-44d0-94d1-4256d5841a2e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:40:46.749727 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2f671b9-bb4c-44d0-94d1-4256d5841a2e None admin] REQUEST_METHOD: `PUT` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:40:46.750269 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2f671b9-bb4c-44d0-94d1-4256d5841a2e None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:40:46.750839 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2f671b9-bb4c-44d0-94d1-4256d5841a2e None admin] PATH_INFO: `/v3/domains/default/users/e5cb815a43fc4460b6ddf0175f087128/roles/f08d6043261747e3b4763c744eef7dfb` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:40:46.754105 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2f671b9-bb4c-44d0-94d1-4256d5841a2e None admin] RBAC: Authorizing `identity:create_grant(domain_id=default, user_id=e5cb815a43fc4460b6ddf0175f087128, role_id=f08d6043261747e3b4763c744eef7dfb)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:40:46.755787 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2f671b9-bb4c-44d0-94d1-4256d5841a2e None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:40:46.774111 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 25/44] 10.222.0.42 () {58 vars in 1361 bytes} [Sun Aug 28 00:40:46 2022] PUT /identity/v3/domains/default/users/e5cb815a43fc4460b6ddf0175f087128/roles/f08d6043261747e3b4763c744eef7dfb => generated 0 bytes in 280 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:40:46.793865 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:40:46.839253 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de30f32d-b418-4288-8829-0aacfa613466 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:40:46.884132 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89d1adcc-5026-41a1-84d2-74755e11b6fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:40:46.884802 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d1adcc-5026-41a1-84d2-74755e11b6fb None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:40:46.885136 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d1adcc-5026-41a1-84d2-74755e11b6fb None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:40:46.885403 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d1adcc-5026-41a1-84d2-74755e11b6fb None admin] PATH_INFO: `/v3/domains/default` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:40:46.886658 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89d1adcc-5026-41a1-84d2-74755e11b6fb None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:40:46.888173 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89d1adcc-5026-41a1-84d2-74755e11b6fb None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:40:46.889921 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 20/45] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:40:46 2022] GET /identity/v3/domains/default => generated 199 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:40:46.896098 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2f671b9-bb4c-44d0-94d1-4256d5841a2e None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:40:46.935993 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:40:47.006166 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2f671b9-bb4c-44d0-94d1-4256d5841a2e None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:40:47.040128 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f27b9507-bace-466c-bc05-c650e1ebc180 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:40:47.040751 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f27b9507-bace-466c-bc05-c650e1ebc180 None admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:40:47.041064 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f27b9507-bace-466c-bc05-c650e1ebc180 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:40:47.041340 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f27b9507-bace-466c-bc05-c650e1ebc180 None admin] PATH_INFO: `/v3/projects` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:40:47.042000 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f27b9507-bace-466c-bc05-c650e1ebc180 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:40:47.043484 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f27b9507-bace-466c-bc05-c650e1ebc180 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:40:47.073277 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 26/46] 10.222.0.42 () {60 vars in 1141 bytes} [Sun Aug 28 00:40:46 2022] POST /identity/v3/projects => generated 312 bytes in 178 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:40:48.699267 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a3a31fe-08d3-4b49-ba43-6c4fba101ea1 None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:40:48.699859 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a3a31fe-08d3-4b49-ba43-6c4fba101ea1 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:40:48.699859 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a3a31fe-08d3-4b49-ba43-6c4fba101ea1 None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:40:48.701225 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 21/47] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:40:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:40:48.707041 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3640f668-39a6-4a6f-b48e-ca4012e18eb6 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:40:48.707439 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3640f668-39a6-4a6f-b48e-ca4012e18eb6 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:40:48.707599 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3640f668-39a6-4a6f-b48e-ca4012e18eb6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:40:48.726355 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-3640f668-39a6-4a6f-b48e-ca4012e18eb6 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:40:48.727558 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-3640f668-39a6-4a6f-b48e-ca4012e18eb6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:40:48.797677 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 27/48] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:40:48 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 28 00:40:48.982920 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8c129a9-273c-4be2-a7d1-3bb55b075273 None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:40:48.983091 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8c129a9-273c-4be2-a7d1-3bb55b075273 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:40:48.983454 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8c129a9-273c-4be2-a7d1-3bb55b075273 None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:40:48.985037 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 22/49] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:40:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:40:48.990329 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f725610-97b5-4a42-b976-ae02ccd3f75e None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:40:48.990622 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f725610-97b5-4a42-b976-ae02ccd3f75e None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:40:48.990891 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f725610-97b5-4a42-b976-ae02ccd3f75e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:40:49.009788 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-9f725610-97b5-4a42-b976-ae02ccd3f75e None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:40:49.010972 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-9f725610-97b5-4a42-b976-ae02ccd3f75e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:40:49.022002 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:40:49.066792 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8c129a9-273c-4be2-a7d1-3bb55b075273 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:40:49.082717 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 28/50] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:40:48 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 28 00:40:49.280094 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8c129a9-273c-4be2-a7d1-3bb55b075273 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:40:49.315052 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28573e8d-c829-4728-8bbd-b0550ff24bb1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:40:49.315689 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28573e8d-c829-4728-8bbd-b0550ff24bb1 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:40:49.315973 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28573e8d-c829-4728-8bbd-b0550ff24bb1 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:40:49.316250 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28573e8d-c829-4728-8bbd-b0550ff24bb1 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:40:49.320158 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:40:49.328656 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28573e8d-c829-4728-8bbd-b0550ff24bb1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 28 00:40:49.335504 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-28573e8d-c829-4728-8bbd-b0550ff24bb1 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 28 00:40:49.335504 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:40:49.335504 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:40:49.335504 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:40:49.335504 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:40:49.335504 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:40:49.335504 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:40:49.335504 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:40:49.335504 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:40:49.335504 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:40:49.335504 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:40:49.335504 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:40:49.335504 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 28 00:40:49.335504 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 28 00:40:49.335504 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 28 00:40:49.335504 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 28 00:40:49.336383 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 28 00:40:49.336383 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 28 00:40:49.336383 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 28 00:40:49.336383 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 28 00:40:49.336383 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:40:49.336383 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:40:49.336383 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:40:49.336383 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:40:49.336383 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:40:49.336383 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:40:49.336383 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:40:49.336383 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:40:49.336383 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:40:49.336383 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:40:49.336383 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:40:49.336383 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:40:49.337285 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:40:49.337285 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:40:49.337285 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:40:49.337285 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:40:49.337285 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 28 00:40:49.337285 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 28 00:40:49.337285 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 28 00:40:49.337285 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 28 00:40:49.337285 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 28 00:40:49.337285 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 28 00:40:49.337285 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 28 00:40:49.337285 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:40:49.337285 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 23/51] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Aug 28 00:40:49 2022] GET /identity/v3/roles/admin => generated 83 bytes in 270 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:40:49.342615 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f725610-97b5-4a42-b976-ae02ccd3f75e None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:40:49.348144 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28573e8d-c829-4728-8bbd-b0550ff24bb1 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:40:49.440533 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f725610-97b5-4a42-b976-ae02ccd3f75e None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:40:49.447937 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28573e8d-c829-4728-8bbd-b0550ff24bb1 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:40:49.474398 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e57aec4-0f3d-4d16-94a6-be16808a7b88 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:40:49.475107 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e57aec4-0f3d-4d16-94a6-be16808a7b88 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:40:49.475392 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e57aec4-0f3d-4d16-94a6-be16808a7b88 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:40:49.475755 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e57aec4-0f3d-4d16-94a6-be16808a7b88 None admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:40:49.476500 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e57aec4-0f3d-4d16-94a6-be16808a7b88 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:40:49.476822 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e57aec4-0f3d-4d16-94a6-be16808a7b88 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:40:49.478445 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e57aec4-0f3d-4d16-94a6-be16808a7b88 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:40:49.482817 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72cb019d-1aee-4325-adb4-ccb0d8906245 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:40:49.483464 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72cb019d-1aee-4325-adb4-ccb0d8906245 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:40:49.483753 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72cb019d-1aee-4325-adb4-ccb0d8906245 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:40:49.484031 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72cb019d-1aee-4325-adb4-ccb0d8906245 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:40:49.485224 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72cb019d-1aee-4325-adb4-ccb0d8906245 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:40:49.486682 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72cb019d-1aee-4325-adb4-ccb0d8906245 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:40:49.486940 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 29/52] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Aug 28 00:40:49 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:40:49.488250 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 24/53] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:40:49 2022] GET /identity/v3/domains/default => generated 199 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:40:49.493485 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e57aec4-0f3d-4d16-94a6-be16808a7b88 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:40:49.493731 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72cb019d-1aee-4325-adb4-ccb0d8906245 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:40:49.505349 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:40:49.559819 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:40:49.600047 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e57aec4-0f3d-4d16-94a6-be16808a7b88 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:40:49.601105 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72cb019d-1aee-4325-adb4-ccb0d8906245 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:40:49.636158 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61900e1d-2d73-4bf9-94ea-b3d23838e955 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:40:49.636803 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61900e1d-2d73-4bf9-94ea-b3d23838e955 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:40:49.637119 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61900e1d-2d73-4bf9-94ea-b3d23838e955 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:40:49.637397 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61900e1d-2d73-4bf9-94ea-b3d23838e955 None admin] PATH_INFO: `/v3/users/e5cb815a43fc4460b6ddf0175f087128` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:40:49.638524 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61900e1d-2d73-4bf9-94ea-b3d23838e955 None admin] RBAC: Authorizing `identity:get_user(user_id=e5cb815a43fc4460b6ddf0175f087128)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:40:49.638943 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8749e608-2804-4ef6-8137-c35a4c8a91ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:40:49.639569 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8749e608-2804-4ef6-8137-c35a4c8a91ff None admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:40:49.639856 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8749e608-2804-4ef6-8137-c35a4c8a91ff None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:40:49.640015 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61900e1d-2d73-4bf9-94ea-b3d23838e955 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:40:49.640160 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8749e608-2804-4ef6-8137-c35a4c8a91ff None admin] PATH_INFO: `/v3/projects` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:40:49.640823 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8749e608-2804-4ef6-8137-c35a4c8a91ff None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:40:49.641691 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 30/54] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Aug 28 00:40:49 2022] GET /identity/v3/users/e5cb815a43fc4460b6ddf0175f087128 => generated 247 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:40:49.642298 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8749e608-2804-4ef6-8137-c35a4c8a91ff None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:40:49.647225 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61900e1d-2d73-4bf9-94ea-b3d23838e955 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:40:49.670121 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 25/55] 10.222.0.42 () {60 vars in 1140 bytes} [Sun Aug 28 00:40:49 2022] POST /identity/v3/projects => generated 298 bytes in 177 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 28 00:40:49.731577 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:40:49.757865 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61900e1d-2d73-4bf9-94ea-b3d23838e955 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:40:49.794227 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcf06f1e-3b36-4139-8c99-b4e692ad38e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:40:49.794860 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf06f1e-3b36-4139-8c99-b4e692ad38e2 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:40:49.795126 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf06f1e-3b36-4139-8c99-b4e692ad38e2 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:40:49.795418 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf06f1e-3b36-4139-8c99-b4e692ad38e2 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:40:49.796628 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcf06f1e-3b36-4139-8c99-b4e692ad38e2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:40:49.798205 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcf06f1e-3b36-4139-8c99-b4e692ad38e2 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:40:49.800022 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 31/56] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:40:49 2022] GET /identity/v3/domains/default => generated 199 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 28 00:40:49.806183 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8749e608-2804-4ef6-8137-c35a4c8a91ff None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:40:49.913444 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8749e608-2804-4ef6-8137-c35a4c8a91ff None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:40:49.933132 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:40:49.955385 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bfab9aa-2e7a-436b-a619-b522f552d70a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:40:49.956055 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bfab9aa-2e7a-436b-a619-b522f552d70a None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:40:49.956331 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bfab9aa-2e7a-436b-a619-b522f552d70a None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:40:49.956600 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bfab9aa-2e7a-436b-a619-b522f552d70a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:40:49.957375 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bfab9aa-2e7a-436b-a619-b522f552d70a None admin] RBAC: Adding query filter params (role.id=f08d6043261747e3b4763c744eef7dfb, scope.domain.id=default, user.id=e5cb815a43fc4460b6ddf0175f087128) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:40:49.957683 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bfab9aa-2e7a-436b-a619-b522f552d70a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:40:49.959198 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bfab9aa-2e7a-436b-a619-b522f552d70a None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:40:49.977072 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 26/57] 10.222.0.42 () {56 vars in 1322 bytes} [Sun Aug 28 00:40:49 2022] GET /identity/v3/role_assignments?user.id=e5cb815a43fc4460b6ddf0175f087128&scope.domain.id=default&role.id=f08d6043261747e3b4763c744eef7dfb => generated 530 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:40:51.579393 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c1ee1b1-8afa-43be-bd7a-4ef5688aa69f None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:40:51.580257 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c1ee1b1-8afa-43be-bd7a-4ef5688aa69f None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:40:51.580257 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c1ee1b1-8afa-43be-bd7a-4ef5688aa69f None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:40:51.581673 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 32/58] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:40:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:40:51.587215 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b169444-b71e-4840-8dc6-7f92acbf38e2 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:40:51.587667 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b169444-b71e-4840-8dc6-7f92acbf38e2 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:40:51.587755 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b169444-b71e-4840-8dc6-7f92acbf38e2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:40:51.614527 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-3b169444-b71e-4840-8dc6-7f92acbf38e2 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:40:51.616214 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-3b169444-b71e-4840-8dc6-7f92acbf38e2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:40:51.690907 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 27/59] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:40:51 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 28 00:40:51.967402 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c1ee1b1-8afa-43be-bd7a-4ef5688aa69f None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:40:52.075658 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c1ee1b1-8afa-43be-bd7a-4ef5688aa69f None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:40:52.110564 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7f3e03c-6568-4f29-bb57-620208e1daff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:40:52.111229 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7f3e03c-6568-4f29-bb57-620208e1daff None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:40:52.111512 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7f3e03c-6568-4f29-bb57-620208e1daff None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:40:52.112226 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7f3e03c-6568-4f29-bb57-620208e1daff None admin] PATH_INFO: `/v3/domains/default` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:40:52.113461 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7f3e03c-6568-4f29-bb57-620208e1daff None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:40:52.114928 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7f3e03c-6568-4f29-bb57-620208e1daff None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:40:52.116521 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 33/60] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:40:51 2022] GET /identity/v3/domains/default => generated 199 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:40:52.122324 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b169444-b71e-4840-8dc6-7f92acbf38e2 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:40:52.227228 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b169444-b71e-4840-8dc6-7f92acbf38e2 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:40:52.238349 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:40:52.263358 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a88032e-0d58-4994-9a02-bc7941baa974 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:40:52.263972 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a88032e-0d58-4994-9a02-bc7941baa974 None admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:40:52.264252 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a88032e-0d58-4994-9a02-bc7941baa974 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:40:52.264528 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a88032e-0d58-4994-9a02-bc7941baa974 None admin] PATH_INFO: `/v3/users` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:40:52.265216 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a88032e-0d58-4994-9a02-bc7941baa974 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:40:52.266729 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a88032e-0d58-4994-9a02-bc7941baa974 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:40:52.295616 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 28/61] 10.222.0.42 () {60 vars in 1132 bytes} [Sun Aug 28 00:40:52 2022] POST /identity/v3/users => generated 275 bytes in 174 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:40:55.283651 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51d34890-ce89-4a3d-9c6f-9de087e05d14 None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:40:55.284796 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51d34890-ce89-4a3d-9c6f-9de087e05d14 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:40:55.285427 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51d34890-ce89-4a3d-9c6f-9de087e05d14 None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:40:55.287688 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 34/62] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:40:55 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:40:55.294090 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da3eae90-327e-4c85-aab8-64068baa0fab None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:40:55.295159 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da3eae90-327e-4c85-aab8-64068baa0fab None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:40:55.295758 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da3eae90-327e-4c85-aab8-64068baa0fab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:40:55.331814 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-da3eae90-327e-4c85-aab8-64068baa0fab None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:40:55.333866 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-da3eae90-327e-4c85-aab8-64068baa0fab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:40:55.473304 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 29/63] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:40:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 28 00:40:55.717265 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37d28d3d-74e5-4166-8814-8b6baec099ba None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:40:55.717938 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37d28d3d-74e5-4166-8814-8b6baec099ba None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:40:55.718415 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37d28d3d-74e5-4166-8814-8b6baec099ba None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:40:55.720103 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 35/64] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:40:55 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:40:55.726453 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa54f760-ec6b-45ff-ab11-a26138c8aeaf None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:40:55.727287 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa54f760-ec6b-45ff-ab11-a26138c8aeaf None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:40:55.727983 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa54f760-ec6b-45ff-ab11-a26138c8aeaf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:40:55.758027 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-fa54f760-ec6b-45ff-ab11-a26138c8aeaf None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:40:55.759774 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-fa54f760-ec6b-45ff-ab11-a26138c8aeaf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:40:55.803357 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e5fe1a0-e208-47b9-9ae1-14b7d6f2a049 None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:40:55.804149 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e5fe1a0-e208-47b9-9ae1-14b7d6f2a049 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:40:55.805019 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e5fe1a0-e208-47b9-9ae1-14b7d6f2a049 None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:40:55.808082 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 36/65] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:40:55 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:40:55.827495 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-155fb53f-effd-45f5-be29-2b344ec5cd20 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:40:55.827779 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-155fb53f-effd-45f5-be29-2b344ec5cd20 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:40:55.828218 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-155fb53f-effd-45f5-be29-2b344ec5cd20 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:40:55.855607 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 30/66] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:40:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 28 00:40:55.857249 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6a90669-5f6a-4166-b28a-cb54c4bff54c None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:40:55.857677 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6a90669-5f6a-4166-b28a-cb54c4bff54c None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:40:55.858111 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6a90669-5f6a-4166-b28a-cb54c4bff54c None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:40:55.858234 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-155fb53f-effd-45f5-be29-2b344ec5cd20 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:40:55.859462 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-155fb53f-effd-45f5-be29-2b344ec5cd20 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:40:55.859974 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 31/67] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:40:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:40:55.866614 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-586aeaed-35de-4ba0-bb53-c92715ed3e39 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:40:55.867095 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-586aeaed-35de-4ba0-bb53-c92715ed3e39 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:40:55.867613 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-586aeaed-35de-4ba0-bb53-c92715ed3e39 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:40:55.876988 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:40:55.900093 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-586aeaed-35de-4ba0-bb53-c92715ed3e39 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:40:55.901360 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-586aeaed-35de-4ba0-bb53-c92715ed3e39 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:40:55.926589 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:40:55.971364 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 37/68] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:40:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 28 00:40:55.973187 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-155fb53f-effd-45f5-be29-2b344ec5cd20 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:40:55.980403 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 32/69] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:40:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 28 00:40:56.093813 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67f3040f-fdb0-4161-b7c2-fc3585ca947a None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:40:56.094499 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67f3040f-fdb0-4161-b7c2-fc3585ca947a None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:40:56.095134 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67f3040f-fdb0-4161-b7c2-fc3585ca947a None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:40:56.095995 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-155fb53f-effd-45f5-be29-2b344ec5cd20 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:40:56.097002 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 33/70] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:40:56 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 28 00:40:56.102313 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-954b91b9-a02c-4935-abe5-928bf66370be None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:40:56.102849 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-954b91b9-a02c-4935-abe5-928bf66370be None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:40:56.103326 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-954b91b9-a02c-4935-abe5-928bf66370be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:40:56.138069 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-314d551f-9909-4638-9d52-d4ba26ec0a2e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:40:56.138983 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-314d551f-9909-4638-9d52-d4ba26ec0a2e None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:40:56.139397 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-314d551f-9909-4638-9d52-d4ba26ec0a2e None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:40:56.139879 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-314d551f-9909-4638-9d52-d4ba26ec0a2e None admin] PATH_INFO: `/v3/roles/member` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:40:56.151625 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-314d551f-9909-4638-9d52-d4ba26ec0a2e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 28 00:40:56.156911 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-314d551f-9909-4638-9d52-d4ba26ec0a2e None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 28 00:40:56.156911 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:40:56.156911 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:40:56.156911 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:40:56.156911 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:40:56.156911 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:40:56.156911 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:40:56.156911 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:40:56.156911 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:40:56.156911 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:40:56.156911 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:40:56.156911 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:40:56.156911 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 28 00:40:56.156911 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 28 00:40:56.156911 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 28 00:40:56.156911 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 28 00:40:56.157773 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 28 00:40:56.157773 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 28 00:40:56.157773 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 28 00:40:56.157773 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 28 00:40:56.157773 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:40:56.157773 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:40:56.157773 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:40:56.157773 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:40:56.157773 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:40:56.157773 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:40:56.157773 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:40:56.157773 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:40:56.157773 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:40:56.157773 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:40:56.157773 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:40:56.157773 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:40:56.158514 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:40:56.158514 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:40:56.158514 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:40:56.158514 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:40:56.158514 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 28 00:40:56.158514 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 28 00:40:56.158514 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 28 00:40:56.158514 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 28 00:40:56.158514 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 28 00:40:56.158514 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 28 00:40:56.158514 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 28 00:40:56.158514 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:40:56.159092 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-954b91b9-a02c-4935-abe5-928bf66370be None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:40:56.159188 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-954b91b9-a02c-4935-abe5-928bf66370be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:40:56.161033 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 38/71] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Aug 28 00:40:55 2022] GET /identity/v3/roles/member => generated 84 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:40:56.167066 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-314d551f-9909-4638-9d52-d4ba26ec0a2e None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:40:56.171633 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:40:56.267936 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 34/72] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:40:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 28 00:40:56.277288 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-954b91b9-a02c-4935-abe5-928bf66370be None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:40:56.310961 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-314d551f-9909-4638-9d52-d4ba26ec0a2e None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:40:56.351175 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2ee37b3-c274-4359-83e6-aaf5e11a630c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:40:56.352059 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2ee37b3-c274-4359-83e6-aaf5e11a630c None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:40:56.352484 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2ee37b3-c274-4359-83e6-aaf5e11a630c None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:40:56.352758 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2ee37b3-c274-4359-83e6-aaf5e11a630c None admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:40:56.353462 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2ee37b3-c274-4359-83e6-aaf5e11a630c None admin] RBAC: Adding query filter params (name=member) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:40:56.353763 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2ee37b3-c274-4359-83e6-aaf5e11a630c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:40:56.355163 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2ee37b3-c274-4359-83e6-aaf5e11a630c None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:40:56.370069 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 39/73] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Aug 28 00:40:56 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 28 00:40:56.370069 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2ee37b3-c274-4359-83e6-aaf5e11a630c None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:40:56.383962 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:40:56.408648 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-954b91b9-a02c-4935-abe5-928bf66370be None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:40:56.460238 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bc9f341-ce2c-443f-92c9-a2c758068813 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:40:56.461011 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bc9f341-ce2c-443f-92c9-a2c758068813 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:40:56.461468 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bc9f341-ce2c-443f-92c9-a2c758068813 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:40:56.461917 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bc9f341-ce2c-443f-92c9-a2c758068813 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:40:56.474316 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:40:56.480226 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bc9f341-ce2c-443f-92c9-a2c758068813 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 28 00:40:56.491216 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-1bc9f341-ce2c-443f-92c9-a2c758068813 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 28 00:40:56.491216 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:40:56.491216 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:40:56.491216 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:40:56.491216 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:40:56.491216 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:40:56.491216 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:40:56.491216 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:40:56.491216 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:40:56.491216 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:40:56.491216 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:40:56.491216 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:40:56.491216 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 28 00:40:56.491216 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 28 00:40:56.491216 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 28 00:40:56.491216 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 28 00:40:56.492068 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 28 00:40:56.492068 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 28 00:40:56.492068 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 28 00:40:56.492068 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 28 00:40:56.492068 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:40:56.492068 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:40:56.492068 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:40:56.492068 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:40:56.492068 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:40:56.492068 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:40:56.492068 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:40:56.492068 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:40:56.492068 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:40:56.492068 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:40:56.492068 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:40:56.492068 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:40:56.492864 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:40:56.492864 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:40:56.492864 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:40:56.492864 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:40:56.492864 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 28 00:40:56.492864 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 28 00:40:56.492864 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 28 00:40:56.492864 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 28 00:40:56.492864 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 28 00:40:56.492864 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 28 00:40:56.492864 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 28 00:40:56.492864 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:40:56.501418 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 35/74] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Aug 28 00:40:56 2022] GET /identity/v3/roles/admin => generated 83 bytes in 232 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:40:56.502728 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bc9f341-ce2c-443f-92c9-a2c758068813 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:40:56.541891 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2ee37b3-c274-4359-83e6-aaf5e11a630c None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:40:56.578063 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af1e63e4-98ac-4546-ba74-a344e7e42a42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:40:56.578775 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af1e63e4-98ac-4546-ba74-a344e7e42a42 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:40:56.579055 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af1e63e4-98ac-4546-ba74-a344e7e42a42 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:40:56.579327 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af1e63e4-98ac-4546-ba74-a344e7e42a42 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:40:56.580736 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af1e63e4-98ac-4546-ba74-a344e7e42a42 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:40:56.582225 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af1e63e4-98ac-4546-ba74-a344e7e42a42 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:40:56.583792 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 40/75] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:40:56 2022] GET /identity/v3/domains/default => generated 199 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:40:56.584544 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af1e63e4-98ac-4546-ba74-a344e7e42a42 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:40:56.619753 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bc9f341-ce2c-443f-92c9-a2c758068813 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:40:56.654826 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b2fedce-6255-4057-8a2b-ef811c4207d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:40:56.655474 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b2fedce-6255-4057-8a2b-ef811c4207d3 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:40:56.655755 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b2fedce-6255-4057-8a2b-ef811c4207d3 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:40:56.656022 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b2fedce-6255-4057-8a2b-ef811c4207d3 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:40:56.667556 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b2fedce-6255-4057-8a2b-ef811c4207d3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 28 00:40:56.672747 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-2b2fedce-6255-4057-8a2b-ef811c4207d3 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 28 00:40:56.672747 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:40:56.672747 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:40:56.672747 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:40:56.672747 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:40:56.672747 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:40:56.672747 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:40:56.672747 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:40:56.672747 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:40:56.672747 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:40:56.672747 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:40:56.672747 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:40:56.672747 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 28 00:40:56.672747 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 28 00:40:56.672747 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 28 00:40:56.672747 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 28 00:40:56.673649 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 28 00:40:56.673649 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 28 00:40:56.673649 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 28 00:40:56.673649 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 28 00:40:56.673649 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:40:56.673649 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:40:56.673649 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:40:56.673649 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:40:56.673649 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:40:56.673649 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:40:56.673649 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:40:56.673649 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:40:56.673649 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:40:56.673649 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:40:56.673649 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:40:56.673649 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:40:56.674429 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:40:56.674429 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:40:56.674429 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:40:56.674429 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:40:56.674429 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 28 00:40:56.674429 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 28 00:40:56.674429 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 28 00:40:56.674429 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 28 00:40:56.674429 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 28 00:40:56.674429 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 28 00:40:56.674429 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 28 00:40:56.674429 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:40:56.674429 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 36/76] 10.222.0.42 () {56 vars in 1114 bytes} [Sun Aug 28 00:40:56 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:40:56.677824 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b2fedce-6255-4057-8a2b-ef811c4207d3 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:40:56.687641 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:40:56.697050 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af1e63e4-98ac-4546-ba74-a344e7e42a42 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:40:56.738116 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:40:56.743983 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0d6a954-1d00-4821-9564-5dfa0d269b91 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:40:56.744617 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0d6a954-1d00-4821-9564-5dfa0d269b91 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:40:56.744895 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0d6a954-1d00-4821-9564-5dfa0d269b91 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:40:56.745194 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0d6a954-1d00-4821-9564-5dfa0d269b91 None admin] PATH_INFO: `/v3/users/196ac391d55640f99b97d658e79fbe0d` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:40:56.758646 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0d6a954-1d00-4821-9564-5dfa0d269b91 None admin] RBAC: Authorizing `identity:get_user(user_id=196ac391d55640f99b97d658e79fbe0d)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:40:56.760103 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0d6a954-1d00-4821-9564-5dfa0d269b91 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:40:56.761781 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 41/77] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Aug 28 00:40:56 2022] GET /identity/v3/users/196ac391d55640f99b97d658e79fbe0d => generated 275 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:40:56.762534 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0d6a954-1d00-4821-9564-5dfa0d269b91 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:40:56.781214 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b2fedce-6255-4057-8a2b-ef811c4207d3 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:40:56.816092 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f61e7827-0afc-4d71-a1d5-832a2f769148 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:40:56.816621 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f61e7827-0afc-4d71-a1d5-832a2f769148 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:40:56.816928 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f61e7827-0afc-4d71-a1d5-832a2f769148 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:40:56.817227 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f61e7827-0afc-4d71-a1d5-832a2f769148 None admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:40:56.817886 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f61e7827-0afc-4d71-a1d5-832a2f769148 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:40:56.818180 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f61e7827-0afc-4d71-a1d5-832a2f769148 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:40:56.819577 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f61e7827-0afc-4d71-a1d5-832a2f769148 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:40:56.827586 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 37/78] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Aug 28 00:40:56 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:40:56.828370 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f61e7827-0afc-4d71-a1d5-832a2f769148 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:40:56.866368 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0d6a954-1d00-4821-9564-5dfa0d269b91 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:40:56.893828 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:40:56.901262 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66f67c6a-cdb0-4adf-bdf4-3cce5c852d71 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:40:56.901910 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66f67c6a-cdb0-4adf-bdf4-3cce5c852d71 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:40:56.902194 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66f67c6a-cdb0-4adf-bdf4-3cce5c852d71 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:40:56.902469 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66f67c6a-cdb0-4adf-bdf4-3cce5c852d71 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:40:56.912793 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66f67c6a-cdb0-4adf-bdf4-3cce5c852d71 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 28 00:40:56.917766 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-66f67c6a-cdb0-4adf-bdf4-3cce5c852d71 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 28 00:40:56.917766 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:40:56.917766 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:40:56.917766 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:40:56.917766 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:40:56.917766 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:40:56.917766 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:40:56.917766 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:40:56.917766 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:40:56.917766 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:40:56.917766 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:40:56.917766 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:40:56.917766 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 28 00:40:56.917766 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 28 00:40:56.917766 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 28 00:40:56.917766 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 28 00:40:56.918715 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 28 00:40:56.918715 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 28 00:40:56.918715 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 28 00:40:56.918715 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 28 00:40:56.918715 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:40:56.918715 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:40:56.918715 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:40:56.918715 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:40:56.918715 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:40:56.918715 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:40:56.918715 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:40:56.918715 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:40:56.918715 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:40:56.918715 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:40:56.918715 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:40:56.918715 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:40:56.919455 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:40:56.919455 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:40:56.919455 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:40:56.919455 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:40:56.919455 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 28 00:40:56.919455 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 28 00:40:56.919455 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 28 00:40:56.919455 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 28 00:40:56.919455 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 28 00:40:56.919455 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 28 00:40:56.919455 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 28 00:40:56.919455 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:40:56.919455 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 42/79] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Aug 28 00:40:56 2022] GET /identity/v3/roles/member => generated 84 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:40:56.920060 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66f67c6a-cdb0-4adf-bdf4-3cce5c852d71 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:40:56.930584 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f61e7827-0afc-4d71-a1d5-832a2f769148 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:40:56.963817 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-505e0118-1ca0-41d4-b52a-4cdfc48f4ef5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:40:56.964432 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-505e0118-1ca0-41d4-b52a-4cdfc48f4ef5 None admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:40:56.964711 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-505e0118-1ca0-41d4-b52a-4cdfc48f4ef5 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:40:56.965002 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-505e0118-1ca0-41d4-b52a-4cdfc48f4ef5 None admin] PATH_INFO: `/v3/users` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:40:56.965658 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-505e0118-1ca0-41d4-b52a-4cdfc48f4ef5 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:40:56.967100 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-505e0118-1ca0-41d4-b52a-4cdfc48f4ef5 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:40:56.970678 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:40:56.989572 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 38/80] 10.222.0.42 () {60 vars in 1132 bytes} [Sun Aug 28 00:40:56 2022] POST /identity/v3/users => generated 289 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:40:56.990729 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-505e0118-1ca0-41d4-b52a-4cdfc48f4ef5 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:40:57.036394 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66f67c6a-cdb0-4adf-bdf4-3cce5c852d71 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:40:57.080842 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-157cb37c-723f-47fd-a1bf-511046b76e2f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:40:57.081476 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-157cb37c-723f-47fd-a1bf-511046b76e2f None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:40:57.081755 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-157cb37c-723f-47fd-a1bf-511046b76e2f None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:40:57.082027 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-157cb37c-723f-47fd-a1bf-511046b76e2f None admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:40:57.082701 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-157cb37c-723f-47fd-a1bf-511046b76e2f None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:40:57.083001 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-157cb37c-723f-47fd-a1bf-511046b76e2f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:40:57.084499 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-157cb37c-723f-47fd-a1bf-511046b76e2f None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:40:57.098286 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 43/81] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Aug 28 00:40:56 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:40:57.098286 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-157cb37c-723f-47fd-a1bf-511046b76e2f None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:40:57.142049 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-505e0118-1ca0-41d4-b52a-4cdfc48f4ef5 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:40:57.169752 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:40:57.186110 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-387f04c3-c4df-4c88-bb5a-c8267d5d0218 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:40:57.187016 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-387f04c3-c4df-4c88-bb5a-c8267d5d0218 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:40:57.187145 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-387f04c3-c4df-4c88-bb5a-c8267d5d0218 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:40:57.187416 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-387f04c3-c4df-4c88-bb5a-c8267d5d0218 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:40:57.215443 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-157cb37c-723f-47fd-a1bf-511046b76e2f None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:40:57.231067 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:40:57.264058 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef72e095-c969-4cb2-a4ca-18e3c310ec0b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:40:57.264708 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef72e095-c969-4cb2-a4ca-18e3c310ec0b None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:40:57.265033 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef72e095-c969-4cb2-a4ca-18e3c310ec0b None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:40:57.265319 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef72e095-c969-4cb2-a4ca-18e3c310ec0b None admin] PATH_INFO: `/v3/users/e5cb815a43fc4460b6ddf0175f087128` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:40:57.266707 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef72e095-c969-4cb2-a4ca-18e3c310ec0b None admin] RBAC: Authorizing `identity:get_user(user_id=e5cb815a43fc4460b6ddf0175f087128)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:40:57.268191 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef72e095-c969-4cb2-a4ca-18e3c310ec0b None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:40:57.270068 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 44/82] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Aug 28 00:40:57 2022] GET /identity/v3/users/e5cb815a43fc4460b6ddf0175f087128 => generated 247 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:40:57.270852 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef72e095-c969-4cb2-a4ca-18e3c310ec0b None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:40:57.276896 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-387f04c3-c4df-4c88-bb5a-c8267d5d0218 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:40:57.278510 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-387f04c3-c4df-4c88-bb5a-c8267d5d0218 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:40:57.400084 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef72e095-c969-4cb2-a4ca-18e3c310ec0b None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:40:57.427046 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 39/83] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Aug 28 00:40:56 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 437 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 28 00:40:57.433369 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-387f04c3-c4df-4c88-bb5a-c8267d5d0218 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:40:57.448265 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e390f66e-eff0-4906-899a-a816f70141c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:40:57.449179 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e390f66e-eff0-4906-899a-a816f70141c3 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:40:57.449495 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e390f66e-eff0-4906-899a-a816f70141c3 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:40:57.450209 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e390f66e-eff0-4906-899a-a816f70141c3 None admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:40:57.451162 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e390f66e-eff0-4906-899a-a816f70141c3 None admin] RBAC: Adding query filter params (name=member) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:40:57.451467 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e390f66e-eff0-4906-899a-a816f70141c3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:40:57.453563 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e390f66e-eff0-4906-899a-a816f70141c3 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:40:57.460701 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:40:57.469156 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 45/84] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Aug 28 00:40:57 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:40:57.470200 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e390f66e-eff0-4906-899a-a816f70141c3 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:40:57.534115 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:40:57.549551 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-387f04c3-c4df-4c88-bb5a-c8267d5d0218 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:40:57.585953 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e46e892-431f-4ac6-b56e-d90e17b1ebd4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:40:57.586654 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e46e892-431f-4ac6-b56e-d90e17b1ebd4 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:40:57.586937 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e46e892-431f-4ac6-b56e-d90e17b1ebd4 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:40:57.587211 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e46e892-431f-4ac6-b56e-d90e17b1ebd4 None admin] PATH_INFO: `/v3/users/196ac391d55640f99b97d658e79fbe0d` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:40:57.588382 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e46e892-431f-4ac6-b56e-d90e17b1ebd4 None admin] RBAC: Authorizing `identity:get_user(user_id=196ac391d55640f99b97d658e79fbe0d)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:40:57.590034 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e46e892-431f-4ac6-b56e-d90e17b1ebd4 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:40:57.591709 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 40/85] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Aug 28 00:40:57 2022] GET /identity/v3/users/196ac391d55640f99b97d658e79fbe0d => generated 275 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:40:57.592539 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e46e892-431f-4ac6-b56e-d90e17b1ebd4 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:40:57.613457 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e390f66e-eff0-4906-899a-a816f70141c3 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:40:57.658908 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54a1663d-5b0e-4fa6-8c94-3d26a02017e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:40:57.659703 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a1663d-5b0e-4fa6-8c94-3d26a02017e5 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:40:57.659994 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a1663d-5b0e-4fa6-8c94-3d26a02017e5 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:40:57.660504 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a1663d-5b0e-4fa6-8c94-3d26a02017e5 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:40:57.705633 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:40:57.725812 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e46e892-431f-4ac6-b56e-d90e17b1ebd4 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:40:57.758260 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54a1663d-5b0e-4fa6-8c94-3d26a02017e5 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:40:57.761478 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54a1663d-5b0e-4fa6-8c94-3d26a02017e5 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:40:57.762471 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a86511f1-7eb4-4acc-89eb-9df158396454 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:40:57.763113 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a86511f1-7eb4-4acc-89eb-9df158396454 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:40:57.763394 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a86511f1-7eb4-4acc-89eb-9df158396454 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:40:57.763665 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a86511f1-7eb4-4acc-89eb-9df158396454 None admin] PATH_INFO: `/v3/projects/39243933b7be4190b983169b99d79165` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:40:57.764936 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a86511f1-7eb4-4acc-89eb-9df158396454 None admin] RBAC: Authorizing `identity:get_project(project_id=39243933b7be4190b983169b99d79165)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:40:57.766516 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a86511f1-7eb4-4acc-89eb-9df158396454 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:40:57.768362 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 41/86] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Aug 28 00:40:57 2022] GET /identity/v3/projects/39243933b7be4190b983169b99d79165 => generated 298 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:40:57.769063 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a86511f1-7eb4-4acc-89eb-9df158396454 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:40:57.844654 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:40:57.877723 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a86511f1-7eb4-4acc-89eb-9df158396454 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:40:57.908445 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 46/87] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Aug 28 00:40:57 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 439 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 28 00:40:57.909230 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54a1663d-5b0e-4fa6-8c94-3d26a02017e5 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:40:57.918621 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b54cda00-1ad3-4527-bef5-6464bb2bfb4e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:40:57.919270 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b54cda00-1ad3-4527-bef5-6464bb2bfb4e None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:40:57.919554 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b54cda00-1ad3-4527-bef5-6464bb2bfb4e None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:40:57.919929 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b54cda00-1ad3-4527-bef5-6464bb2bfb4e None admin] PATH_INFO: `/v3/users/196ac391d55640f99b97d658e79fbe0d` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:40:57.921130 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b54cda00-1ad3-4527-bef5-6464bb2bfb4e None admin] RBAC: Authorizing `identity:get_user(user_id=196ac391d55640f99b97d658e79fbe0d)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:40:57.922738 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b54cda00-1ad3-4527-bef5-6464bb2bfb4e None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:40:57.924591 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 42/88] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Aug 28 00:40:57 2022] GET /identity/v3/users/196ac391d55640f99b97d658e79fbe0d => generated 275 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:40:57.929878 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b54cda00-1ad3-4527-bef5-6464bb2bfb4e None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:40:57.961517 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:40:58.022339 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54a1663d-5b0e-4fa6-8c94-3d26a02017e5 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:40:58.039704 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b54cda00-1ad3-4527-bef5-6464bb2bfb4e None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:40:58.061721 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8539942-7a79-4627-997e-ae519d399a22 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:40:58.062394 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8539942-7a79-4627-997e-ae519d399a22 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:40:58.062672 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8539942-7a79-4627-997e-ae519d399a22 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:40:58.063026 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8539942-7a79-4627-997e-ae519d399a22 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:40:58.076252 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:40:58.076495 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f31744d-34f1-42e3-8a39-6570394fc192 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:40:58.077165 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f31744d-34f1-42e3-8a39-6570394fc192 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:40:58.077449 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f31744d-34f1-42e3-8a39-6570394fc192 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:40:58.077834 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f31744d-34f1-42e3-8a39-6570394fc192 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:40:58.078456 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f31744d-34f1-42e3-8a39-6570394fc192 None admin] RBAC: Adding query filter params (role.id=0487ef7e943e4701915cb8ce0f2899d7, scope.project.id=39243933b7be4190b983169b99d79165, user.id=196ac391d55640f99b97d658e79fbe0d) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:40:58.078874 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f31744d-34f1-42e3-8a39-6570394fc192 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:40:58.080626 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f31744d-34f1-42e3-8a39-6570394fc192 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:40:58.089186 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 43/89] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Aug 28 00:40:57 2022] GET /identity/v3/role_assignments?user.id=196ac391d55640f99b97d658e79fbe0d&scope.project.id=39243933b7be4190b983169b99d79165&role.id=0487ef7e943e4701915cb8ce0f2899d7 => generated 259 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:40:58.090138 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f31744d-34f1-42e3-8a39-6570394fc192 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:40:58.174657 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8539942-7a79-4627-997e-ae519d399a22 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:40:58.176120 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8539942-7a79-4627-997e-ae519d399a22 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:40:58.201716 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:40:58.206773 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f31744d-34f1-42e3-8a39-6570394fc192 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:40:58.245982 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63b2ef50-ae86-47c1-890f-9660b19b4f92 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:40:58.247081 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63b2ef50-ae86-47c1-890f-9660b19b4f92 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:40:58.247600 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63b2ef50-ae86-47c1-890f-9660b19b4f92 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:40:58.248091 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63b2ef50-ae86-47c1-890f-9660b19b4f92 None admin] PATH_INFO: `/v3/projects/39243933b7be4190b983169b99d79165` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:40:58.249592 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63b2ef50-ae86-47c1-890f-9660b19b4f92 None admin] RBAC: Authorizing `identity:get_project(project_id=39243933b7be4190b983169b99d79165)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:40:58.251581 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63b2ef50-ae86-47c1-890f-9660b19b4f92 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:40:58.253763 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 44/90] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Aug 28 00:40:58 2022] GET /identity/v3/projects/39243933b7be4190b983169b99d79165 => generated 298 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:40:58.254949 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63b2ef50-ae86-47c1-890f-9660b19b4f92 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:40:58.364323 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 47/91] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Aug 28 00:40:57 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 455 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 28 00:40:58.364831 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63b2ef50-ae86-47c1-890f-9660b19b4f92 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:40:58.365292 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8539942-7a79-4627-997e-ae519d399a22 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:40:58.425419 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:40:58.429906 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b107b00-4d65-4738-bee3-7a6fd0c6682a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:40:58.431459 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b107b00-4d65-4738-bee3-7a6fd0c6682a None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:40:58.431739 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b107b00-4d65-4738-bee3-7a6fd0c6682a None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:40:58.432028 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b107b00-4d65-4738-bee3-7a6fd0c6682a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:40:58.491138 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:40:58.540506 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8539942-7a79-4627-997e-ae519d399a22 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:40:58.574162 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b107b00-4d65-4738-bee3-7a6fd0c6682a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:40:58.575670 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b107b00-4d65-4738-bee3-7a6fd0c6682a None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:40:58.615571 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8975a97-0516-4048-8e14-74bd9decb8f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:40:58.616197 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8975a97-0516-4048-8e14-74bd9decb8f4 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:40:58.616472 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8975a97-0516-4048-8e14-74bd9decb8f4 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:40:58.616741 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8975a97-0516-4048-8e14-74bd9decb8f4 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:40:58.617780 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8975a97-0516-4048-8e14-74bd9decb8f4 None admin] RBAC: Adding query filter params (role.id=f08d6043261747e3b4763c744eef7dfb, scope.project.id=39243933b7be4190b983169b99d79165, user.id=e5cb815a43fc4460b6ddf0175f087128) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:40:58.618087 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8975a97-0516-4048-8e14-74bd9decb8f4 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:40:58.619953 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8975a97-0516-4048-8e14-74bd9decb8f4 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:40:58.628846 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 48/92] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Aug 28 00:40:58 2022] GET /identity/v3/role_assignments?user.id=e5cb815a43fc4460b6ddf0175f087128&scope.project.id=39243933b7be4190b983169b99d79165&role.id=f08d6043261747e3b4763c744eef7dfb => generated 259 bytes in 264 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:40:58.629709 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8975a97-0516-4048-8e14-74bd9decb8f4 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:40:58.751715 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 45/93] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Aug 28 00:40:58 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 497 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 28 00:40:58.757367 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b107b00-4d65-4738-bee3-7a6fd0c6682a None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:40:58.820403 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:40:58.854539 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8975a97-0516-4048-8e14-74bd9decb8f4 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:40:58.864535 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b107b00-4d65-4738-bee3-7a6fd0c6682a None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:40:58.867940 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:40:58.902963 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1836114-abd3-4064-9113-01a0bb906f4a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:40:58.903659 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1836114-abd3-4064-9113-01a0bb906f4a None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:40:58.903914 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1836114-abd3-4064-9113-01a0bb906f4a None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:40:58.904184 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1836114-abd3-4064-9113-01a0bb906f4a None admin] PATH_INFO: `/v3/projects/5341d633667247b9b075a2fcbe4c0156` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:40:58.905362 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1836114-abd3-4064-9113-01a0bb906f4a None admin] RBAC: Authorizing `identity:get_project(project_id=5341d633667247b9b075a2fcbe4c0156)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:40:58.912985 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1836114-abd3-4064-9113-01a0bb906f4a None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:40:58.912985 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 46/94] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Aug 28 00:40:58 2022] GET /identity/v3/projects/5341d633667247b9b075a2fcbe4c0156 => generated 312 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 28 00:40:58.916167 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1836114-abd3-4064-9113-01a0bb906f4a None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:40:58.944996 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c57aceb-2176-4392-ba9c-0bc12d5b20ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:40:58.945924 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c57aceb-2176-4392-ba9c-0bc12d5b20ee None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:40:58.947080 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c57aceb-2176-4392-ba9c-0bc12d5b20ee None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:40:58.947626 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c57aceb-2176-4392-ba9c-0bc12d5b20ee None admin] PATH_INFO: `/v3/projects/39243933b7be4190b983169b99d79165` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:40:58.950447 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c57aceb-2176-4392-ba9c-0bc12d5b20ee None admin] RBAC: Authorizing `identity:get_project(project_id=39243933b7be4190b983169b99d79165)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:40:58.953741 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c57aceb-2176-4392-ba9c-0bc12d5b20ee None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:40:58.961014 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 49/95] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Aug 28 00:40:58 2022] GET /identity/v3/projects/39243933b7be4190b983169b99d79165 => generated 298 bytes in 331 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:40:58.963972 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c57aceb-2176-4392-ba9c-0bc12d5b20ee None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:40:59.140875 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c57aceb-2176-4392-ba9c-0bc12d5b20ee None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:40:59.148794 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1836114-abd3-4064-9113-01a0bb906f4a None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:40:59.178597 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:40:59.186581 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddae78b3-3582-4e5d-8aba-3282b3be6b5c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:40:59.187222 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddae78b3-3582-4e5d-8aba-3282b3be6b5c None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:40:59.187508 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddae78b3-3582-4e5d-8aba-3282b3be6b5c None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:40:59.187785 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddae78b3-3582-4e5d-8aba-3282b3be6b5c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:40:59.188613 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddae78b3-3582-4e5d-8aba-3282b3be6b5c None admin] RBAC: Adding query filter params (role.id=0487ef7e943e4701915cb8ce0f2899d7, scope.project.id=5341d633667247b9b075a2fcbe4c0156, user.id=196ac391d55640f99b97d658e79fbe0d) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:40:59.189318 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddae78b3-3582-4e5d-8aba-3282b3be6b5c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:40:59.191143 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddae78b3-3582-4e5d-8aba-3282b3be6b5c None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:40:59.200990 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 47/96] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Aug 28 00:40:58 2022] GET /identity/v3/role_assignments?user.id=196ac391d55640f99b97d658e79fbe0d&scope.project.id=5341d633667247b9b075a2fcbe4c0156&role.id=0487ef7e943e4701915cb8ce0f2899d7 => generated 259 bytes in 283 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:40:59.228377 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fa190cb-3919-4df4-94dd-130edcec35c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:40:59.229332 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fa190cb-3919-4df4-94dd-130edcec35c1 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:40:59.229753 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fa190cb-3919-4df4-94dd-130edcec35c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:40:59.230161 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fa190cb-3919-4df4-94dd-130edcec35c1 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:40:59.231068 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fa190cb-3919-4df4-94dd-130edcec35c1 None admin] RBAC: Adding query filter params (role.id=05caf0846a15452b9a6a79fa59c7fa5a, scope.project.id=39243933b7be4190b983169b99d79165, user.id=196ac391d55640f99b97d658e79fbe0d) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:40:59.231584 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fa190cb-3919-4df4-94dd-130edcec35c1 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:40:59.233339 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fa190cb-3919-4df4-94dd-130edcec35c1 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:40:59.245130 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 50/97] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Aug 28 00:40:58 2022] GET /identity/v3/role_assignments?user.id=196ac391d55640f99b97d658e79fbe0d&scope.project.id=39243933b7be4190b983169b99d79165&role.id=05caf0846a15452b9a6a79fa59c7fa5a => generated 259 bytes in 282 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:40:59.360803 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c32569f4-7601-46ad-8a25-daf3cf0c2379 None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:40:59.361142 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c32569f4-7601-46ad-8a25-daf3cf0c2379 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:40:59.361429 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c32569f4-7601-46ad-8a25-daf3cf0c2379 None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:40:59.363044 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 48/98] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:40:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:40:59.368938 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9371699-0ead-44b1-9b71-4aafc1e28240 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:40:59.369230 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9371699-0ead-44b1-9b71-4aafc1e28240 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:40:59.369458 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9371699-0ead-44b1-9b71-4aafc1e28240 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:40:59.390020 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-d9371699-0ead-44b1-9b71-4aafc1e28240 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:40:59.391301 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-d9371699-0ead-44b1-9b71-4aafc1e28240 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:40:59.427543 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:40:59.473451 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 51/99] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:40:59 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 28 00:40:59.594102 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82df4945-0dbb-46b0-9123-d00689a226b0 None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:40:59.594603 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82df4945-0dbb-46b0-9123-d00689a226b0 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:40:59.594711 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82df4945-0dbb-46b0-9123-d00689a226b0 None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:40:59.596297 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 49/100] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:40:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:40:59.601735 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f53d554-c9af-4e21-9ce1-277012b33bf5 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:40:59.602028 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f53d554-c9af-4e21-9ce1-277012b33bf5 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:40:59.602294 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f53d554-c9af-4e21-9ce1-277012b33bf5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:40:59.630170 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-3f53d554-c9af-4e21-9ce1-277012b33bf5 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:40:59.631468 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-3f53d554-c9af-4e21-9ce1-277012b33bf5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:40:59.723242 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 52/101] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:40:59 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 28 00:40:59.968987 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82df4945-0dbb-46b0-9123-d00689a226b0 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:00.132082 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82df4945-0dbb-46b0-9123-d00689a226b0 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:00.168671 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe10b699-2ab7-4709-9c55-813ffd902c96 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:00.169346 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe10b699-2ab7-4709-9c55-813ffd902c96 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:00.169624 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe10b699-2ab7-4709-9c55-813ffd902c96 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:00.169895 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe10b699-2ab7-4709-9c55-813ffd902c96 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:00.188299 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe10b699-2ab7-4709-9c55-813ffd902c96 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 28 00:41:00.193423 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-fe10b699-2ab7-4709-9c55-813ffd902c96 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 28 00:41:00.193423 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:41:00.193423 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:41:00.193423 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:41:00.193423 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:41:00.193423 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:41:00.193423 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:41:00.193423 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:41:00.193423 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:41:00.193423 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:41:00.193423 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:41:00.193423 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:41:00.193423 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 28 00:41:00.193423 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 28 00:41:00.193423 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 28 00:41:00.193423 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 28 00:41:00.194404 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 28 00:41:00.194404 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 28 00:41:00.194404 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 28 00:41:00.194404 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 28 00:41:00.194404 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:41:00.194404 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:41:00.194404 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:41:00.194404 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:41:00.194404 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:41:00.194404 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:41:00.194404 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:41:00.194404 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:41:00.194404 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:41:00.194404 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:41:00.194404 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:41:00.194404 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:41:00.195160 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:41:00.195160 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:41:00.195160 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:41:00.195160 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:41:00.195160 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 28 00:41:00.195160 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 28 00:41:00.195160 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 28 00:41:00.195160 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 28 00:41:00.195160 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 28 00:41:00.195160 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 28 00:41:00.195160 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 28 00:41:00.195160 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:41:00.195160 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 50/102] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Aug 28 00:40:59 2022] GET /identity/v3/roles/reader => generated 84 bytes in 226 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:41:00.200792 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f53d554-c9af-4e21-9ce1-277012b33bf5 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:00.303822 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe10b699-2ab7-4709-9c55-813ffd902c96 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:00.351352 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:00.359412 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f53d554-c9af-4e21-9ce1-277012b33bf5 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:00.397358 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:00.444590 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24ca8edb-cd20-46e7-9bf8-3008eb0d63c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:00.445275 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24ca8edb-cd20-46e7-9bf8-3008eb0d63c9 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:00.445517 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24ca8edb-cd20-46e7-9bf8-3008eb0d63c9 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:00.445789 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24ca8edb-cd20-46e7-9bf8-3008eb0d63c9 None admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:00.446527 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24ca8edb-cd20-46e7-9bf8-3008eb0d63c9 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:00.446832 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24ca8edb-cd20-46e7-9bf8-3008eb0d63c9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:00.448236 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24ca8edb-cd20-46e7-9bf8-3008eb0d63c9 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:00.481135 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 53/103] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Aug 28 00:41:00 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 281 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:00.497505 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24ca8edb-cd20-46e7-9bf8-3008eb0d63c9 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:00.533437 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe10b699-2ab7-4709-9c55-813ffd902c96 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:00.629745 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4ce8216-fd6d-41ab-a2a3-ff77eceebaec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:00.631440 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4ce8216-fd6d-41ab-a2a3-ff77eceebaec None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:00.631728 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4ce8216-fd6d-41ab-a2a3-ff77eceebaec None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:00.632003 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4ce8216-fd6d-41ab-a2a3-ff77eceebaec None admin] PATH_INFO: `/v3/domains/default` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:00.633877 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4ce8216-fd6d-41ab-a2a3-ff77eceebaec None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:00.635346 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4ce8216-fd6d-41ab-a2a3-ff77eceebaec None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:00.636997 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 51/104] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:41:00 2022] GET /identity/v3/domains/default => generated 199 bytes in 334 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 28 00:41:00.645326 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4ce8216-fd6d-41ab-a2a3-ff77eceebaec None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:00.751290 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24ca8edb-cd20-46e7-9bf8-3008eb0d63c9 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:00.808728 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4ce8216-fd6d-41ab-a2a3-ff77eceebaec None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:00.829354 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:00.844209 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cd255b3-b037-429c-a8d1-e5712f1c55e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:00.844838 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cd255b3-b037-429c-a8d1-e5712f1c55e6 None admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:00.845252 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cd255b3-b037-429c-a8d1-e5712f1c55e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:00.845537 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cd255b3-b037-429c-a8d1-e5712f1c55e6 None admin] PATH_INFO: `/v3/projects` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:00.845697 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d115d748-9f0d-4ba4-bce8-53b0bc0a214d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:00.846216 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cd255b3-b037-429c-a8d1-e5712f1c55e6 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:00.846330 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d115d748-9f0d-4ba4-bce8-53b0bc0a214d None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:00.846612 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d115d748-9f0d-4ba4-bce8-53b0bc0a214d None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:00.846890 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d115d748-9f0d-4ba4-bce8-53b0bc0a214d None admin] PATH_INFO: `/v3/users/b1c830143c754b80a0b378a0ad4b96a5` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:00.847802 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cd255b3-b037-429c-a8d1-e5712f1c55e6 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:00.879718 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 52/105] 10.222.0.42 () {60 vars in 1141 bytes} [Sun Aug 28 00:41:00 2022] POST /identity/v3/projects => generated 302 bytes in 235 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:41:00.881354 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b154096c-1346-4f02-8e23-d9e25d6f878d None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:00.881590 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b154096c-1346-4f02-8e23-d9e25d6f878d None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:00.881869 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b154096c-1346-4f02-8e23-d9e25d6f878d None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:00.883269 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 53/106] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:41:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:41:00.886122 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d115d748-9f0d-4ba4-bce8-53b0bc0a214d None admin] RBAC: Authorizing `identity:get_user(user_id=b1c830143c754b80a0b378a0ad4b96a5)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:00.887671 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d115d748-9f0d-4ba4-bce8-53b0bc0a214d None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:00.889719 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 54/107] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Aug 28 00:41:00 2022] GET /identity/v3/users/b1c830143c754b80a0b378a0ad4b96a5 => generated 289 bytes in 393 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 28 00:41:00.894367 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcfc0917-ff23-4a08-acdf-ba486053c21b None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:00.894664 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcfc0917-ff23-4a08-acdf-ba486053c21b None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:00.894938 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcfc0917-ff23-4a08-acdf-ba486053c21b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:00.895133 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d115d748-9f0d-4ba4-bce8-53b0bc0a214d None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:00.917084 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-bcfc0917-ff23-4a08-acdf-ba486053c21b None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:41:00.919224 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-bcfc0917-ff23-4a08-acdf-ba486053c21b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:41:00.954256 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:00.999843 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 54/108] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:41:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 28 00:41:01.120976 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d115d748-9f0d-4ba4-bce8-53b0bc0a214d None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:01.179293 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68a0daf2-50dc-4892-9c9f-2c3ce37720d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:01.179842 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a0daf2-50dc-4892-9c9f-2c3ce37720d7 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:01.180128 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a0daf2-50dc-4892-9c9f-2c3ce37720d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:01.180409 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a0daf2-50dc-4892-9c9f-2c3ce37720d7 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:01.272534 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68a0daf2-50dc-4892-9c9f-2c3ce37720d7 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:01.274130 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68a0daf2-50dc-4892-9c9f-2c3ce37720d7 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:01.302154 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:01.352019 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcfc0917-ff23-4a08-acdf-ba486053c21b None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:01.453475 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 55/109] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Aug 28 00:41:00 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 559 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Aug 28 00:41:01.459428 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68a0daf2-50dc-4892-9c9f-2c3ce37720d7 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:01.473208 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcfc0917-ff23-4a08-acdf-ba486053c21b None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:01.475028 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:01.513159 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42aab0ec-e328-4a45-8419-070bd24c66b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:01.513819 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42aab0ec-e328-4a45-8419-070bd24c66b1 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:01.514102 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42aab0ec-e328-4a45-8419-070bd24c66b1 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:01.514537 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42aab0ec-e328-4a45-8419-070bd24c66b1 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:01.526190 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42aab0ec-e328-4a45-8419-070bd24c66b1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 28 00:41:01.531954 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-42aab0ec-e328-4a45-8419-070bd24c66b1 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 28 00:41:01.531954 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:41:01.531954 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:41:01.531954 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:41:01.531954 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:41:01.531954 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:41:01.531954 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:41:01.531954 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:41:01.531954 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:41:01.531954 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:41:01.531954 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:41:01.531954 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:41:01.531954 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 28 00:41:01.531954 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 28 00:41:01.531954 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 28 00:41:01.531954 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 28 00:41:01.532823 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 28 00:41:01.532823 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 28 00:41:01.532823 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 28 00:41:01.532823 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 28 00:41:01.532823 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:41:01.532823 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:41:01.532823 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:41:01.532823 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:41:01.532823 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:41:01.532823 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:41:01.532823 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:41:01.532823 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:41:01.532823 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:41:01.532823 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:41:01.532823 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:41:01.532823 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:41:01.533617 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:41:01.533617 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:41:01.533617 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:41:01.533617 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:41:01.533617 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 28 00:41:01.533617 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 28 00:41:01.533617 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 28 00:41:01.533617 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 28 00:41:01.533617 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 28 00:41:01.533617 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 28 00:41:01.533617 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 28 00:41:01.533617 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:41:01.533617 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 55/110] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Aug 28 00:41:01 2022] GET /identity/v3/roles/member => generated 84 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 28 00:41:01.539383 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42aab0ec-e328-4a45-8419-070bd24c66b1 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:01.648984 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42aab0ec-e328-4a45-8419-070bd24c66b1 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:01.688267 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-591aa450-10e5-49a4-b8c6-c35e0f91f70e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:01.688900 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-591aa450-10e5-49a4-b8c6-c35e0f91f70e None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:01.689369 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-591aa450-10e5-49a4-b8c6-c35e0f91f70e None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:01.689659 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-591aa450-10e5-49a4-b8c6-c35e0f91f70e None admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:01.690521 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-591aa450-10e5-49a4-b8c6-c35e0f91f70e None admin] RBAC: Adding query filter params (name=member) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:01.690839 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-591aa450-10e5-49a4-b8c6-c35e0f91f70e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:01.692282 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-591aa450-10e5-49a4-b8c6-c35e0f91f70e None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:01.707157 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 56/111] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Aug 28 00:41:01 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:01.708568 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-591aa450-10e5-49a4-b8c6-c35e0f91f70e None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:01.737461 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:01.752234 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68a0daf2-50dc-4892-9c9f-2c3ce37720d7 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:01.773009 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:01.787436 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac170630-7022-4008-9312-32abbfd1a03e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:01.788083 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac170630-7022-4008-9312-32abbfd1a03e None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:01.788362 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac170630-7022-4008-9312-32abbfd1a03e None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:01.788636 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac170630-7022-4008-9312-32abbfd1a03e None admin] PATH_INFO: `/v3/projects/39243933b7be4190b983169b99d79165` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:01.789843 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac170630-7022-4008-9312-32abbfd1a03e None admin] RBAC: Authorizing `identity:get_project(project_id=39243933b7be4190b983169b99d79165)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:01.791323 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac170630-7022-4008-9312-32abbfd1a03e None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:01.792994 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 56/112] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Aug 28 00:41:01 2022] GET /identity/v3/projects/39243933b7be4190b983169b99d79165 => generated 298 bytes in 334 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 28 00:41:01.794166 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db0099b0-312d-461f-9240-6f8aaa9538f7 None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:01.794446 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db0099b0-312d-461f-9240-6f8aaa9538f7 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:01.794718 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db0099b0-312d-461f-9240-6f8aaa9538f7 None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:01.796194 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 57/113] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:41:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:41:01.801442 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b37f0ce7-b28a-460d-9e2f-83265293f83d None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:01.801786 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b37f0ce7-b28a-460d-9e2f-83265293f83d None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:01.802007 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b37f0ce7-b28a-460d-9e2f-83265293f83d None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:01.803477 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 58/114] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:41:01 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:41:01.804235 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b37f0ce7-b28a-460d-9e2f-83265293f83d None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:01.827469 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-591aa450-10e5-49a4-b8c6-c35e0f91f70e None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:01.872131 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebabb009-19ad-47e8-b484-b526f5513491 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:01.872838 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebabb009-19ad-47e8-b484-b526f5513491 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:01.873153 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebabb009-19ad-47e8-b484-b526f5513491 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:01.873472 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebabb009-19ad-47e8-b484-b526f5513491 None admin] PATH_INFO: `/v3/users/196ac391d55640f99b97d658e79fbe0d` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:01.874672 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebabb009-19ad-47e8-b484-b526f5513491 None admin] RBAC: Authorizing `identity:get_user(user_id=196ac391d55640f99b97d658e79fbe0d)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:01.876213 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebabb009-19ad-47e8-b484-b526f5513491 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:01.878188 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 57/115] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Aug 28 00:41:01 2022] GET /identity/v3/users/196ac391d55640f99b97d658e79fbe0d => generated 275 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 28 00:41:01.879263 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99c23c5a-3333-47de-be8f-0813af086f1d None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:01.879542 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99c23c5a-3333-47de-be8f-0813af086f1d None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:01.879862 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99c23c5a-3333-47de-be8f-0813af086f1d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:01.898834 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-99c23c5a-3333-47de-be8f-0813af086f1d None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:41:01.900029 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-99c23c5a-3333-47de-be8f-0813af086f1d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:41:01.927876 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b37f0ce7-b28a-460d-9e2f-83265293f83d None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:01.964352 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8591c28a-80cf-4f13-8bc1-d6291b39aaa1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:01.964892 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8591c28a-80cf-4f13-8bc1-d6291b39aaa1 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:01.965202 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8591c28a-80cf-4f13-8bc1-d6291b39aaa1 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:01.965473 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8591c28a-80cf-4f13-8bc1-d6291b39aaa1 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:01.966212 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8591c28a-80cf-4f13-8bc1-d6291b39aaa1 None admin] RBAC: Adding query filter params (role.id=60823fd2c27d46e98b5d299e86b51918, scope.project.id=39243933b7be4190b983169b99d79165, user.id=b1c830143c754b80a0b378a0ad4b96a5) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:01.966520 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8591c28a-80cf-4f13-8bc1-d6291b39aaa1 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:01.967986 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8591c28a-80cf-4f13-8bc1-d6291b39aaa1 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:01.970422 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 58/116] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:41:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 28 00:41:01.972006 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76244645-0ae7-44c4-9572-0d79ec2db363 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:01.972467 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76244645-0ae7-44c4-9572-0d79ec2db363 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:01.973060 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76244645-0ae7-44c4-9572-0d79ec2db363 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:01.974421 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 59/117] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Aug 28 00:41:01 2022] GET /identity/v3/role_assignments?user.id=b1c830143c754b80a0b378a0ad4b96a5&scope.project.id=39243933b7be4190b983169b99d79165&role.id=60823fd2c27d46e98b5d299e86b51918 => generated 259 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:01.975390 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8591c28a-80cf-4f13-8bc1-d6291b39aaa1 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:01.993431 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-76244645-0ae7-44c4-9572-0d79ec2db363 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:41:01.995670 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-76244645-0ae7-44c4-9572-0d79ec2db363 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:41:02.029431 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:02.055282 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:02.080131 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 59/118] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:41:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 28 00:41:02.104163 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d64bd875-2b52-4135-a7f7-40c0ac770fd2 None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:02.104849 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d64bd875-2b52-4135-a7f7-40c0ac770fd2 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:02.105337 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d64bd875-2b52-4135-a7f7-40c0ac770fd2 None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:02.107224 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 60/119] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:41:02 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:41:02.127150 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a4d59ef-edbf-42ec-b2c0-c9b4f36afdc1 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:02.127783 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a4d59ef-edbf-42ec-b2c0-c9b4f36afdc1 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:02.128246 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a4d59ef-edbf-42ec-b2c0-c9b4f36afdc1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:02.134243 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8591c28a-80cf-4f13-8bc1-d6291b39aaa1 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:02.162192 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-2a4d59ef-edbf-42ec-b2c0-c9b4f36afdc1 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:41:02.163814 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-2a4d59ef-edbf-42ec-b2c0-c9b4f36afdc1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:41:02.169002 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63cef4ca-e517-420e-aa68-9b6c89a82e44 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:02.169654 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63cef4ca-e517-420e-aa68-9b6c89a82e44 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:02.169938 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63cef4ca-e517-420e-aa68-9b6c89a82e44 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:02.170215 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63cef4ca-e517-420e-aa68-9b6c89a82e44 None admin] PATH_INFO: `/v3/projects/39243933b7be4190b983169b99d79165` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:02.171373 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63cef4ca-e517-420e-aa68-9b6c89a82e44 None admin] RBAC: Authorizing `identity:get_project(project_id=39243933b7be4190b983169b99d79165)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:02.177020 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63cef4ca-e517-420e-aa68-9b6c89a82e44 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:02.177020 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 60/120] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Aug 28 00:41:01 2022] GET /identity/v3/projects/39243933b7be4190b983169b99d79165 => generated 298 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:02.180658 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63cef4ca-e517-420e-aa68-9b6c89a82e44 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:02.282113 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 61/121] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:41:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 28 00:41:02.291926 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63cef4ca-e517-420e-aa68-9b6c89a82e44 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:02.315177 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:02.334299 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fadf5d4-3ff8-4850-a78e-438c8a96bdb6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:02.335039 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fadf5d4-3ff8-4850-a78e-438c8a96bdb6 None admin] REQUEST_METHOD: `PUT` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:02.335327 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fadf5d4-3ff8-4850-a78e-438c8a96bdb6 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:02.335608 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fadf5d4-3ff8-4850-a78e-438c8a96bdb6 None admin] PATH_INFO: `/v3/projects/39243933b7be4190b983169b99d79165/users/196ac391d55640f99b97d658e79fbe0d/roles/0487ef7e943e4701915cb8ce0f2899d7` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:02.337697 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fadf5d4-3ff8-4850-a78e-438c8a96bdb6 None admin] RBAC: Authorizing `identity:create_grant(project_id=39243933b7be4190b983169b99d79165, user_id=196ac391d55640f99b97d658e79fbe0d, role_id=0487ef7e943e4701915cb8ce0f2899d7)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:02.339169 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fadf5d4-3ff8-4850-a78e-438c8a96bdb6 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:02.360798 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 61/122] 10.222.0.42 () {58 vars in 1439 bytes} [Sun Aug 28 00:41:02 2022] PUT /identity/v3/projects/39243933b7be4190b983169b99d79165/users/196ac391d55640f99b97d658e79fbe0d/roles/0487ef7e943e4701915cb8ce0f2899d7 => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 28 00:41:02.421468 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a4d59ef-edbf-42ec-b2c0-c9b4f36afdc1 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:02.491412 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fadf5d4-3ff8-4850-a78e-438c8a96bdb6 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:02.517579 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:02.540241 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a4d59ef-edbf-42ec-b2c0-c9b4f36afdc1 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:02.577125 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-124a81c7-d1d0-4e56-8147-8d216b2f17d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:02.577865 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-124a81c7-d1d0-4e56-8147-8d216b2f17d4 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:02.578148 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-124a81c7-d1d0-4e56-8147-8d216b2f17d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:02.578451 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-124a81c7-d1d0-4e56-8147-8d216b2f17d4 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:02.590328 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-124a81c7-d1d0-4e56-8147-8d216b2f17d4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 28 00:41:02.595508 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-124a81c7-d1d0-4e56-8147-8d216b2f17d4 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 28 00:41:02.595508 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:41:02.595508 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:41:02.595508 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:41:02.595508 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:41:02.595508 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:41:02.595508 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:41:02.595508 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:41:02.595508 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:41:02.595508 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:41:02.595508 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:41:02.595508 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:41:02.595508 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 28 00:41:02.595508 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 28 00:41:02.595508 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 28 00:41:02.595508 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 28 00:41:02.596452 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 28 00:41:02.596452 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 28 00:41:02.596452 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 28 00:41:02.596452 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 28 00:41:02.596452 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:41:02.596452 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:41:02.596452 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:41:02.596452 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:41:02.596452 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:41:02.596452 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:41:02.596452 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:41:02.596452 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:41:02.596452 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:41:02.596452 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:41:02.596452 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:41:02.596452 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:41:02.597244 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:41:02.597244 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:41:02.597244 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:41:02.597244 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:41:02.597244 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 28 00:41:02.597244 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 28 00:41:02.597244 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 28 00:41:02.597244 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 28 00:41:02.597244 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 28 00:41:02.597244 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 28 00:41:02.597244 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 28 00:41:02.597244 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:41:02.597244 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 62/123] 10.222.0.42 () {56 vars in 1114 bytes} [Sun Aug 28 00:41:02 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:41:02.605020 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-124a81c7-d1d0-4e56-8147-8d216b2f17d4 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:02.676930 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fadf5d4-3ff8-4850-a78e-438c8a96bdb6 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:02.713457 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f81d5c37-aa9a-40ce-b989-402179c81b68 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:02.714106 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f81d5c37-aa9a-40ce-b989-402179c81b68 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:02.714388 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f81d5c37-aa9a-40ce-b989-402179c81b68 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:02.714661 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f81d5c37-aa9a-40ce-b989-402179c81b68 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:02.727518 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f81d5c37-aa9a-40ce-b989-402179c81b68 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 28 00:41:02.731869 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-124a81c7-d1d0-4e56-8147-8d216b2f17d4 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:02.733976 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-f81d5c37-aa9a-40ce-b989-402179c81b68 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 28 00:41:02.733976 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:41:02.733976 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:41:02.733976 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:41:02.733976 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:41:02.733976 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:41:02.733976 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:41:02.733976 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:41:02.733976 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:41:02.733976 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:41:02.733976 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:41:02.733976 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:41:02.733976 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 28 00:41:02.733976 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 28 00:41:02.733976 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 28 00:41:02.733976 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 28 00:41:02.734891 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 28 00:41:02.734891 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 28 00:41:02.734891 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 28 00:41:02.734891 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 28 00:41:02.734891 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:41:02.734891 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:41:02.734891 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:41:02.734891 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:41:02.734891 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:41:02.734891 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:41:02.734891 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:41:02.734891 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:41:02.734891 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:41:02.734891 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:41:02.734891 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:41:02.734891 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:41:02.735652 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:41:02.735652 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:41:02.735652 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:41:02.735652 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:41:02.735652 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 28 00:41:02.735652 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 28 00:41:02.735652 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 28 00:41:02.735652 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 28 00:41:02.735652 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 28 00:41:02.735652 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 28 00:41:02.735652 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 28 00:41:02.735652 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:41:02.740146 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 62/124] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Aug 28 00:41:02 2022] GET /identity/v3/roles/admin => generated 83 bytes in 245 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:41:02.741538 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f81d5c37-aa9a-40ce-b989-402179c81b68 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:02.768546 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6db3eee-c2af-43a3-b9be-788a4640d8f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:02.769209 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6db3eee-c2af-43a3-b9be-788a4640d8f8 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:02.769495 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6db3eee-c2af-43a3-b9be-788a4640d8f8 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:02.769772 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6db3eee-c2af-43a3-b9be-788a4640d8f8 None admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:02.770565 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6db3eee-c2af-43a3-b9be-788a4640d8f8 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:02.770879 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6db3eee-c2af-43a3-b9be-788a4640d8f8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:02.772329 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6db3eee-c2af-43a3-b9be-788a4640d8f8 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:02.782248 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 63/125] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Aug 28 00:41:02 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:02.786168 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:02.788278 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6db3eee-c2af-43a3-b9be-788a4640d8f8 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:02.805593 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:02.857263 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f81d5c37-aa9a-40ce-b989-402179c81b68 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:02.893232 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-693d3487-9d5c-4d9f-8293-97b053883f32 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:02.894378 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-693d3487-9d5c-4d9f-8293-97b053883f32 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:02.894838 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-693d3487-9d5c-4d9f-8293-97b053883f32 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:02.895250 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-693d3487-9d5c-4d9f-8293-97b053883f32 None admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:02.896062 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-693d3487-9d5c-4d9f-8293-97b053883f32 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:02.896504 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-693d3487-9d5c-4d9f-8293-97b053883f32 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:02.898094 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-693d3487-9d5c-4d9f-8293-97b053883f32 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:02.901584 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6db3eee-c2af-43a3-b9be-788a4640d8f8 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:02.913101 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 63/126] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Aug 28 00:41:02 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:02.914345 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-693d3487-9d5c-4d9f-8293-97b053883f32 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:02.939396 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a3e3400-ee56-4dc7-a585-acf3d2dd1767 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:02.940056 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a3e3400-ee56-4dc7-a585-acf3d2dd1767 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:02.940489 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a3e3400-ee56-4dc7-a585-acf3d2dd1767 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:02.940661 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a3e3400-ee56-4dc7-a585-acf3d2dd1767 None admin] PATH_INFO: `/v3/users/196ac391d55640f99b97d658e79fbe0d` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:02.942002 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a3e3400-ee56-4dc7-a585-acf3d2dd1767 None admin] RBAC: Authorizing `identity:get_user(user_id=196ac391d55640f99b97d658e79fbe0d)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:02.943755 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a3e3400-ee56-4dc7-a585-acf3d2dd1767 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:02.945468 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 64/127] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Aug 28 00:41:02 2022] GET /identity/v3/users/196ac391d55640f99b97d658e79fbe0d => generated 275 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:02.946456 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a3e3400-ee56-4dc7-a585-acf3d2dd1767 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:03.024030 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-693d3487-9d5c-4d9f-8293-97b053883f32 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:03.026054 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:03.061348 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36bd1747-f100-4282-af97-99cf883e3732 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:03.062316 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36bd1747-f100-4282-af97-99cf883e3732 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:03.062893 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36bd1747-f100-4282-af97-99cf883e3732 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:03.063360 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36bd1747-f100-4282-af97-99cf883e3732 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:03.082233 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36bd1747-f100-4282-af97-99cf883e3732 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 28 00:41:03.089376 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:03.094561 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-36bd1747-f100-4282-af97-99cf883e3732 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 28 00:41:03.094561 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:41:03.094561 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:41:03.094561 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:41:03.094561 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:41:03.094561 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:41:03.094561 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:41:03.094561 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:41:03.094561 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:41:03.094561 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:41:03.094561 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:41:03.094561 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:41:03.094561 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 28 00:41:03.094561 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 28 00:41:03.094561 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 28 00:41:03.094561 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 28 00:41:03.095484 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 28 00:41:03.095484 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 28 00:41:03.095484 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 28 00:41:03.095484 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 28 00:41:03.095484 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:41:03.095484 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:41:03.095484 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:41:03.095484 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:41:03.095484 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:41:03.095484 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:41:03.095484 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:41:03.095484 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:41:03.095484 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:41:03.095484 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:41:03.095484 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:41:03.095484 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:41:03.096331 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:41:03.096331 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:41:03.096331 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:41:03.096331 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:41:03.096331 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 28 00:41:03.096331 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 28 00:41:03.096331 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 28 00:41:03.096331 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 28 00:41:03.096331 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 28 00:41:03.096331 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 28 00:41:03.096331 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 28 00:41:03.096331 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:41:03.096331 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 64/128] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Aug 28 00:41:02 2022] GET /identity/v3/roles/member => generated 84 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:41:03.099402 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36bd1747-f100-4282-af97-99cf883e3732 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:03.141424 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a3e3400-ee56-4dc7-a585-acf3d2dd1767 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:03.217814 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc6109db-89c1-4965-ba25-8597fd825783 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:03.218466 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc6109db-89c1-4965-ba25-8597fd825783 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:03.218751 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc6109db-89c1-4965-ba25-8597fd825783 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:03.219097 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc6109db-89c1-4965-ba25-8597fd825783 None admin] PATH_INFO: `/v3/users/e5cb815a43fc4460b6ddf0175f087128` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:03.221752 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc6109db-89c1-4965-ba25-8597fd825783 None admin] RBAC: Authorizing `identity:get_user(user_id=e5cb815a43fc4460b6ddf0175f087128)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:03.225973 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc6109db-89c1-4965-ba25-8597fd825783 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:03.228797 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 65/129] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Aug 28 00:41:02 2022] GET /identity/v3/users/e5cb815a43fc4460b6ddf0175f087128 => generated 247 bytes in 283 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:03.232145 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc6109db-89c1-4965-ba25-8597fd825783 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:03.299723 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36bd1747-f100-4282-af97-99cf883e3732 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:03.365987 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-996873c4-89db-4a48-8610-949d47872c74 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:03.366552 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-996873c4-89db-4a48-8610-949d47872c74 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:03.366886 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-996873c4-89db-4a48-8610-949d47872c74 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:03.367160 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-996873c4-89db-4a48-8610-949d47872c74 None admin] PATH_INFO: `/v3/projects/39243933b7be4190b983169b99d79165` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:03.368319 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-996873c4-89db-4a48-8610-949d47872c74 None admin] RBAC: Authorizing `identity:get_project(project_id=39243933b7be4190b983169b99d79165)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:03.369842 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-996873c4-89db-4a48-8610-949d47872c74 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:03.371720 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 65/130] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Aug 28 00:41:03 2022] GET /identity/v3/projects/39243933b7be4190b983169b99d79165 => generated 298 bytes in 275 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:03.372446 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-996873c4-89db-4a48-8610-949d47872c74 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:03.402119 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:03.424496 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc6109db-89c1-4965-ba25-8597fd825783 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:03.460466 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0947de8-bc7d-47ea-90f2-a31ab099d2a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:03.461125 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0947de8-bc7d-47ea-90f2-a31ab099d2a9 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:03.461413 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0947de8-bc7d-47ea-90f2-a31ab099d2a9 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:03.461685 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0947de8-bc7d-47ea-90f2-a31ab099d2a9 None admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:03.462350 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0947de8-bc7d-47ea-90f2-a31ab099d2a9 None admin] RBAC: Adding query filter params (name=member) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:03.462651 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0947de8-bc7d-47ea-90f2-a31ab099d2a9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:03.464034 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0947de8-bc7d-47ea-90f2-a31ab099d2a9 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:03.473242 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 66/131] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Aug 28 00:41:03 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 244 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:03.474127 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0947de8-bc7d-47ea-90f2-a31ab099d2a9 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:03.478785 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:03.543012 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-996873c4-89db-4a48-8610-949d47872c74 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:03.609080 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0947de8-bc7d-47ea-90f2-a31ab099d2a9 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:03.613969 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-993faed1-d1ec-4562-b1eb-5bc09a5fb853 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:03.614622 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-993faed1-d1ec-4562-b1eb-5bc09a5fb853 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:03.614907 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-993faed1-d1ec-4562-b1eb-5bc09a5fb853 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:03.615192 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-993faed1-d1ec-4562-b1eb-5bc09a5fb853 None admin] PATH_INFO: `/v3/projects/39243933b7be4190b983169b99d79165` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:03.618037 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-993faed1-d1ec-4562-b1eb-5bc09a5fb853 None admin] RBAC: Authorizing `identity:get_project(project_id=39243933b7be4190b983169b99d79165)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:03.619472 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-993faed1-d1ec-4562-b1eb-5bc09a5fb853 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:03.626875 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 66/132] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Aug 28 00:41:03 2022] GET /identity/v3/projects/39243933b7be4190b983169b99d79165 => generated 298 bytes in 255 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:03.627875 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-993faed1-d1ec-4562-b1eb-5bc09a5fb853 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:03.692165 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f4d8289-d99a-4c40-bdcd-625a0bfb2e01 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:03.692854 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f4d8289-d99a-4c40-bdcd-625a0bfb2e01 None admin] REQUEST_METHOD: `PUT` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:03.693163 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f4d8289-d99a-4c40-bdcd-625a0bfb2e01 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:03.693442 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f4d8289-d99a-4c40-bdcd-625a0bfb2e01 None admin] PATH_INFO: `/v3/projects/39243933b7be4190b983169b99d79165/users/196ac391d55640f99b97d658e79fbe0d/roles/05caf0846a15452b9a6a79fa59c7fa5a` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:03.696657 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f4d8289-d99a-4c40-bdcd-625a0bfb2e01 None admin] RBAC: Authorizing `identity:create_grant(project_id=39243933b7be4190b983169b99d79165, user_id=196ac391d55640f99b97d658e79fbe0d, role_id=05caf0846a15452b9a6a79fa59c7fa5a)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:03.702418 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f4d8289-d99a-4c40-bdcd-625a0bfb2e01 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:03.732262 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 67/133] 10.222.0.42 () {58 vars in 1439 bytes} [Sun Aug 28 00:41:03 2022] PUT /identity/v3/projects/39243933b7be4190b983169b99d79165/users/196ac391d55640f99b97d658e79fbe0d/roles/05caf0846a15452b9a6a79fa59c7fa5a => generated 0 bytes in 258 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:41:03.733967 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f4d8289-d99a-4c40-bdcd-625a0bfb2e01 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:03.831025 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:03.847152 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-993faed1-d1ec-4562-b1eb-5bc09a5fb853 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:03.881335 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:03.924514 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-527fb1a4-8f4f-4a01-8178-8fc3ce7b8bf9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:03.925205 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-527fb1a4-8f4f-4a01-8178-8fc3ce7b8bf9 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:03.925492 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-527fb1a4-8f4f-4a01-8178-8fc3ce7b8bf9 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:03.925770 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-527fb1a4-8f4f-4a01-8178-8fc3ce7b8bf9 None admin] PATH_INFO: `/v3/users/196ac391d55640f99b97d658e79fbe0d` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:03.928249 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-527fb1a4-8f4f-4a01-8178-8fc3ce7b8bf9 None admin] RBAC: Authorizing `identity:get_user(user_id=196ac391d55640f99b97d658e79fbe0d)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:03.930730 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-527fb1a4-8f4f-4a01-8178-8fc3ce7b8bf9 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:03.932652 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 67/134] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Aug 28 00:41:03 2022] GET /identity/v3/users/196ac391d55640f99b97d658e79fbe0d => generated 275 bytes in 305 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:03.943218 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-527fb1a4-8f4f-4a01-8178-8fc3ce7b8bf9 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:03.969742 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f4d8289-d99a-4c40-bdcd-625a0bfb2e01 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:04.031040 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-215b1ac6-a062-402c-bdc1-6e88ce8b6790 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:04.031765 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-215b1ac6-a062-402c-bdc1-6e88ce8b6790 None admin] REQUEST_METHOD: `PUT` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:04.032053 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-215b1ac6-a062-402c-bdc1-6e88ce8b6790 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:04.033585 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-215b1ac6-a062-402c-bdc1-6e88ce8b6790 None admin] PATH_INFO: `/v3/projects/39243933b7be4190b983169b99d79165/users/e5cb815a43fc4460b6ddf0175f087128/roles/f08d6043261747e3b4763c744eef7dfb` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:04.037182 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-215b1ac6-a062-402c-bdc1-6e88ce8b6790 None admin] RBAC: Authorizing `identity:create_grant(project_id=39243933b7be4190b983169b99d79165, user_id=e5cb815a43fc4460b6ddf0175f087128, role_id=f08d6043261747e3b4763c744eef7dfb)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:04.039569 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-215b1ac6-a062-402c-bdc1-6e88ce8b6790 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:04.062125 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 68/135] 10.222.0.42 () {58 vars in 1439 bytes} [Sun Aug 28 00:41:03 2022] PUT /identity/v3/projects/39243933b7be4190b983169b99d79165/users/e5cb815a43fc4460b6ddf0175f087128/roles/f08d6043261747e3b4763c744eef7dfb => generated 0 bytes in 329 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:41:04.108646 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-527fb1a4-8f4f-4a01-8178-8fc3ce7b8bf9 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:04.144637 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9daebe2f-d957-4236-aa20-912067617c60 None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:04.144934 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9daebe2f-d957-4236-aa20-912067617c60 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:04.145207 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9daebe2f-d957-4236-aa20-912067617c60 None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:04.146690 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 69/136] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:41:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 28 00:41:04.158805 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-196156bc-6abd-45b2-ae97-50b893f94202 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:04.159136 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-196156bc-6abd-45b2-ae97-50b893f94202 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:04.159373 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-196156bc-6abd-45b2-ae97-50b893f94202 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:04.175676 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1570f91-a283-4e5b-b992-88c3b2537c82 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:04.176336 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1570f91-a283-4e5b-b992-88c3b2537c82 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:04.176620 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1570f91-a283-4e5b-b992-88c3b2537c82 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:04.176900 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1570f91-a283-4e5b-b992-88c3b2537c82 None admin] PATH_INFO: `/v3/projects/5341d633667247b9b075a2fcbe4c0156` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:04.186062 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1570f91-a283-4e5b-b992-88c3b2537c82 None admin] RBAC: Authorizing `identity:get_project(project_id=5341d633667247b9b075a2fcbe4c0156)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:04.187620 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1570f91-a283-4e5b-b992-88c3b2537c82 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:04.189853 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 68/137] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Aug 28 00:41:03 2022] GET /identity/v3/projects/5341d633667247b9b075a2fcbe4c0156 => generated 312 bytes in 247 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 28 00:41:04.190693 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-196156bc-6abd-45b2-ae97-50b893f94202 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:41:04.191926 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-196156bc-6abd-45b2-ae97-50b893f94202 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:41:04.202366 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1570f91-a283-4e5b-b992-88c3b2537c82 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:04.265962 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:04.335507 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 70/138] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:41:04 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 28 00:41:04.393933 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1570f91-a283-4e5b-b992-88c3b2537c82 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:04.445144 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3153110-8de8-4cfc-8148-daae0a0bdf2a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:04.445764 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3153110-8de8-4cfc-8148-daae0a0bdf2a None admin] REQUEST_METHOD: `PUT` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:04.446047 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3153110-8de8-4cfc-8148-daae0a0bdf2a None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:04.446323 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3153110-8de8-4cfc-8148-daae0a0bdf2a None admin] PATH_INFO: `/v3/projects/5341d633667247b9b075a2fcbe4c0156/users/196ac391d55640f99b97d658e79fbe0d/roles/0487ef7e943e4701915cb8ce0f2899d7` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:04.448395 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3153110-8de8-4cfc-8148-daae0a0bdf2a None admin] RBAC: Authorizing `identity:create_grant(project_id=5341d633667247b9b075a2fcbe4c0156, user_id=196ac391d55640f99b97d658e79fbe0d, role_id=0487ef7e943e4701915cb8ce0f2899d7)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:04.449930 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3153110-8de8-4cfc-8148-daae0a0bdf2a None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:04.466738 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 69/139] 10.222.0.42 () {58 vars in 1439 bytes} [Sun Aug 28 00:41:04 2022] PUT /identity/v3/projects/5341d633667247b9b075a2fcbe4c0156/users/196ac391d55640f99b97d658e79fbe0d/roles/0487ef7e943e4701915cb8ce0f2899d7 => generated 0 bytes in 265 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 28 00:41:04.695655 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e1ea794-b0dd-4aa3-97a3-cab71224aa75 None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:04.695989 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e1ea794-b0dd-4aa3-97a3-cab71224aa75 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:04.696161 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e1ea794-b0dd-4aa3-97a3-cab71224aa75 None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:04.697662 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 71/140] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:41:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:41:04.703186 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b029e776-cf3d-48bc-bd49-6fc2f138ef34 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:04.703690 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b029e776-cf3d-48bc-bd49-6fc2f138ef34 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:04.703888 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b029e776-cf3d-48bc-bd49-6fc2f138ef34 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:04.734416 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-b029e776-cf3d-48bc-bd49-6fc2f138ef34 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:41:04.735792 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-b029e776-cf3d-48bc-bd49-6fc2f138ef34 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:41:04.803486 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e1ea794-b0dd-4aa3-97a3-cab71224aa75 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:04.827762 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 70/141] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:41:04 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 28 00:41:04.873351 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:04.928874 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7288ba3-fe4e-40d2-ba2c-279402cae380 None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:04.929225 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7288ba3-fe4e-40d2-ba2c-279402cae380 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:04.929466 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7288ba3-fe4e-40d2-ba2c-279402cae380 None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:04.931551 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 71/142] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:41:04 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 28 00:41:04.937365 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08ddcf64-a4f1-4688-aeb6-0608941a6f8d None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:04.937654 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08ddcf64-a4f1-4688-aeb6-0608941a6f8d None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:04.937969 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08ddcf64-a4f1-4688-aeb6-0608941a6f8d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:04.942817 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e1ea794-b0dd-4aa3-97a3-cab71224aa75 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:04.963214 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-08ddcf64-a4f1-4688-aeb6-0608941a6f8d None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:41:04.964338 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-08ddcf64-a4f1-4688-aeb6-0608941a6f8d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:41:04.989757 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:04.997524 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6531955-50c9-4885-b44c-db7815622b0e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:04.998147 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6531955-50c9-4885-b44c-db7815622b0e None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:04.998426 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6531955-50c9-4885-b44c-db7815622b0e None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:04.998700 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6531955-50c9-4885-b44c-db7815622b0e None admin] PATH_INFO: `/v3/domains/default` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:04.999966 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6531955-50c9-4885-b44c-db7815622b0e None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:05.001792 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6531955-50c9-4885-b44c-db7815622b0e None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:05.003417 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 72/143] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:41:04 2022] GET /identity/v3/domains/default => generated 199 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 28 00:41:05.015122 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6531955-50c9-4885-b44c-db7815622b0e None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:05.158455 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 72/144] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:41:04 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 222 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 28 00:41:05.200624 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6531955-50c9-4885-b44c-db7815622b0e None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:05.235103 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:05.254648 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5345e8d8-a098-4949-ad0e-60cbcaae2c11 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:05.255274 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5345e8d8-a098-4949-ad0e-60cbcaae2c11 None admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:05.255550 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5345e8d8-a098-4949-ad0e-60cbcaae2c11 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:05.255817 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5345e8d8-a098-4949-ad0e-60cbcaae2c11 None admin] PATH_INFO: `/v3/users` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:05.256474 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5345e8d8-a098-4949-ad0e-60cbcaae2c11 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:05.257958 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5345e8d8-a098-4949-ad0e-60cbcaae2c11 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:05.291430 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 73/145] 10.222.0.42 () {60 vars in 1132 bytes} [Sun Aug 28 00:41:05 2022] POST /identity/v3/users => generated 283 bytes in 277 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:41:05.308838 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08ddcf64-a4f1-4688-aeb6-0608941a6f8d None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:05.438898 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08ddcf64-a4f1-4688-aeb6-0608941a6f8d None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:05.492480 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a327160f-50ea-43eb-849c-d20a989862c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:05.493088 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a327160f-50ea-43eb-849c-d20a989862c9 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:05.493369 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a327160f-50ea-43eb-849c-d20a989862c9 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:05.493640 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a327160f-50ea-43eb-849c-d20a989862c9 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:05.509131 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a327160f-50ea-43eb-849c-d20a989862c9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 28 00:41:05.514267 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-a327160f-50ea-43eb-849c-d20a989862c9 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 28 00:41:05.514267 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:41:05.514267 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:41:05.514267 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:41:05.514267 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:41:05.514267 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:41:05.514267 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:41:05.514267 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:41:05.514267 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:41:05.514267 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:41:05.514267 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:41:05.514267 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:41:05.514267 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 28 00:41:05.514267 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 28 00:41:05.514267 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 28 00:41:05.514267 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 28 00:41:05.515237 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 28 00:41:05.515237 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 28 00:41:05.515237 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 28 00:41:05.515237 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 28 00:41:05.515237 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:41:05.515237 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:41:05.515237 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:41:05.515237 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:41:05.515237 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:41:05.515237 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:41:05.515237 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:41:05.515237 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:41:05.515237 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:41:05.515237 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:41:05.515237 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:41:05.515237 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:41:05.516054 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:41:05.516054 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:41:05.516054 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:41:05.516054 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:41:05.516054 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 28 00:41:05.516054 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 28 00:41:05.516054 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 28 00:41:05.516054 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 28 00:41:05.516054 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 28 00:41:05.516054 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 28 00:41:05.516054 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 28 00:41:05.516054 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:41:05.520784 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 73/146] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Aug 28 00:41:05 2022] GET /identity/v3/roles/reader => generated 84 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:41:05.523229 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5345e8d8-a098-4949-ad0e-60cbcaae2c11 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:05.664687 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5345e8d8-a098-4949-ad0e-60cbcaae2c11 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:05.714228 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32e59b5d-4d63-4349-8ed9-ed62750dfbe1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:05.715256 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e59b5d-4d63-4349-8ed9-ed62750dfbe1 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:05.715745 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e59b5d-4d63-4349-8ed9-ed62750dfbe1 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:05.716214 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e59b5d-4d63-4349-8ed9-ed62750dfbe1 None admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:05.717119 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32e59b5d-4d63-4349-8ed9-ed62750dfbe1 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:05.717623 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32e59b5d-4d63-4349-8ed9-ed62750dfbe1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:05.719290 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32e59b5d-4d63-4349-8ed9-ed62750dfbe1 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:05.732636 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 74/147] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Aug 28 00:41:05 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:05.738673 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a327160f-50ea-43eb-849c-d20a989862c9 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:05.761755 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:05.769705 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32e59b5d-4d63-4349-8ed9-ed62750dfbe1 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:05.839311 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:05.907980 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32e59b5d-4d63-4349-8ed9-ed62750dfbe1 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:05.909952 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a327160f-50ea-43eb-849c-d20a989862c9 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:05.949632 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d700641-3b69-472c-aa25-510221080de4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:05.950271 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d700641-3b69-472c-aa25-510221080de4 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:05.950546 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d700641-3b69-472c-aa25-510221080de4 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:05.950813 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d700641-3b69-472c-aa25-510221080de4 None admin] PATH_INFO: `/v3/users/b1c830143c754b80a0b378a0ad4b96a5` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:05.951995 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d700641-3b69-472c-aa25-510221080de4 None admin] RBAC: Authorizing `identity:get_user(user_id=b1c830143c754b80a0b378a0ad4b96a5)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:05.953578 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d700641-3b69-472c-aa25-510221080de4 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:05.955492 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 74/148] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Aug 28 00:41:05 2022] GET /identity/v3/users/b1c830143c754b80a0b378a0ad4b96a5 => generated 289 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:05.957593 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4d4b0ef-e517-4ae8-843d-adb2e08d8944 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:05.958465 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4d4b0ef-e517-4ae8-843d-adb2e08d8944 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:05.958923 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4d4b0ef-e517-4ae8-843d-adb2e08d8944 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:05.959361 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4d4b0ef-e517-4ae8-843d-adb2e08d8944 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:05.961283 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d700641-3b69-472c-aa25-510221080de4 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:05.971350 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4d4b0ef-e517-4ae8-843d-adb2e08d8944 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 28 00:41:05.977302 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-f4d4b0ef-e517-4ae8-843d-adb2e08d8944 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 28 00:41:05.977302 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:41:05.977302 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:41:05.977302 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:41:05.977302 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:41:05.977302 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:41:05.977302 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:41:05.977302 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:41:05.977302 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:41:05.977302 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:41:05.977302 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:41:05.977302 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:41:05.977302 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 28 00:41:05.977302 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 28 00:41:05.977302 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 28 00:41:05.977302 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 28 00:41:05.978185 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 28 00:41:05.978185 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 28 00:41:05.978185 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 28 00:41:05.978185 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 28 00:41:05.978185 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:41:05.978185 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:41:05.978185 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:41:05.978185 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:41:05.978185 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:41:05.978185 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:41:05.978185 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:41:05.978185 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:41:05.978185 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:41:05.978185 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:41:05.978185 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:41:05.978185 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:41:05.978971 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:41:05.978971 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:41:05.978971 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:41:05.978971 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:41:05.978971 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 28 00:41:05.978971 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 28 00:41:05.978971 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 28 00:41:05.978971 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 28 00:41:05.978971 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 28 00:41:05.978971 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 28 00:41:05.978971 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 28 00:41:05.978971 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:41:05.980984 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 75/149] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Aug 28 00:41:05 2022] GET /identity/v3/roles/member => generated 84 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:41:05.991293 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4d4b0ef-e517-4ae8-843d-adb2e08d8944 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:06.103995 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:06.108189 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4d4b0ef-e517-4ae8-843d-adb2e08d8944 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:06.119113 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:06.149199 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d677656f-da9f-471c-85a4-9f7c3206e780 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:06.150065 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d677656f-da9f-471c-85a4-9f7c3206e780 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:06.150484 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d677656f-da9f-471c-85a4-9f7c3206e780 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:06.150896 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d677656f-da9f-471c-85a4-9f7c3206e780 None admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:06.151705 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d677656f-da9f-471c-85a4-9f7c3206e780 None admin] RBAC: Adding query filter params (name=member) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:06.152170 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d677656f-da9f-471c-85a4-9f7c3206e780 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:06.153723 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d677656f-da9f-471c-85a4-9f7c3206e780 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:06.162831 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 76/150] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Aug 28 00:41:05 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 28 00:41:06.163797 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d700641-3b69-472c-aa25-510221080de4 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:06.174117 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d677656f-da9f-471c-85a4-9f7c3206e780 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:06.206343 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-323b9a66-a16a-43f9-840f-1098fc4df4c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:06.206987 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-323b9a66-a16a-43f9-840f-1098fc4df4c3 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:06.207262 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-323b9a66-a16a-43f9-840f-1098fc4df4c3 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:06.207587 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-323b9a66-a16a-43f9-840f-1098fc4df4c3 None admin] PATH_INFO: `/v3/projects/39243933b7be4190b983169b99d79165` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:06.208668 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-323b9a66-a16a-43f9-840f-1098fc4df4c3 None admin] RBAC: Authorizing `identity:get_project(project_id=39243933b7be4190b983169b99d79165)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:06.210717 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-323b9a66-a16a-43f9-840f-1098fc4df4c3 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:06.212386 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 75/151] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Aug 28 00:41:05 2022] GET /identity/v3/projects/39243933b7be4190b983169b99d79165 => generated 298 bytes in 251 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:06.218405 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-323b9a66-a16a-43f9-840f-1098fc4df4c3 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:06.312082 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d677656f-da9f-471c-85a4-9f7c3206e780 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:06.331355 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-323b9a66-a16a-43f9-840f-1098fc4df4c3 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:06.359118 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa7053fe-1a13-41f7-9970-feffa744a560 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:06.360053 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa7053fe-1a13-41f7-9970-feffa744a560 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:06.360476 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa7053fe-1a13-41f7-9970-feffa744a560 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:06.360906 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa7053fe-1a13-41f7-9970-feffa744a560 None admin] PATH_INFO: `/v3/users/196ac391d55640f99b97d658e79fbe0d` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:06.362318 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa7053fe-1a13-41f7-9970-feffa744a560 None admin] RBAC: Authorizing `identity:get_user(user_id=196ac391d55640f99b97d658e79fbe0d)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:06.363940 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa7053fe-1a13-41f7-9970-feffa744a560 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:06.365865 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 77/152] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Aug 28 00:41:06 2022] GET /identity/v3/users/196ac391d55640f99b97d658e79fbe0d => generated 275 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:06.368931 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b9a0cb3-4506-43fc-a0fc-9b6ea349f812 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:06.369632 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b9a0cb3-4506-43fc-a0fc-9b6ea349f812 None admin] REQUEST_METHOD: `PUT` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:06.369910 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b9a0cb3-4506-43fc-a0fc-9b6ea349f812 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:06.370178 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b9a0cb3-4506-43fc-a0fc-9b6ea349f812 None admin] PATH_INFO: `/v3/projects/39243933b7be4190b983169b99d79165/users/b1c830143c754b80a0b378a0ad4b96a5/roles/60823fd2c27d46e98b5d299e86b51918` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:06.371802 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa7053fe-1a13-41f7-9970-feffa744a560 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:06.372177 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b9a0cb3-4506-43fc-a0fc-9b6ea349f812 None admin] RBAC: Authorizing `identity:create_grant(project_id=39243933b7be4190b983169b99d79165, user_id=b1c830143c754b80a0b378a0ad4b96a5, role_id=60823fd2c27d46e98b5d299e86b51918)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:06.373786 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b9a0cb3-4506-43fc-a0fc-9b6ea349f812 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:06.398569 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 76/153] 10.222.0.42 () {58 vars in 1439 bytes} [Sun Aug 28 00:41:06 2022] PUT /identity/v3/projects/39243933b7be4190b983169b99d79165/users/b1c830143c754b80a0b378a0ad4b96a5/roles/60823fd2c27d46e98b5d299e86b51918 => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 28 00:41:06.409696 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:06.515697 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa7053fe-1a13-41f7-9970-feffa744a560 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:06.560809 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19577d25-95ca-4b3b-ace9-d0bd517ccf12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:06.561502 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19577d25-95ca-4b3b-ace9-d0bd517ccf12 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:06.561782 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19577d25-95ca-4b3b-ace9-d0bd517ccf12 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:06.562054 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19577d25-95ca-4b3b-ace9-d0bd517ccf12 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:06.644414 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19577d25-95ca-4b3b-ace9-d0bd517ccf12 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:06.645972 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19577d25-95ca-4b3b-ace9-d0bd517ccf12 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:06.672789 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4962462-eddc-4c79-9f6a-db30deeb33f8 None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:06.673447 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4962462-eddc-4c79-9f6a-db30deeb33f8 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:06.673931 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4962462-eddc-4c79-9f6a-db30deeb33f8 None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:06.675778 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 77/154] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:41:06 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:41:06.681599 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de8e3e3d-7f8e-446e-9fbc-9e201d3f7bcc None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:06.682174 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de8e3e3d-7f8e-446e-9fbc-9e201d3f7bcc None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:06.682658 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de8e3e3d-7f8e-446e-9fbc-9e201d3f7bcc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:06.690630 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:06.725587 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-de8e3e3d-7f8e-446e-9fbc-9e201d3f7bcc None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:41:06.727506 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-de8e3e3d-7f8e-446e-9fbc-9e201d3f7bcc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:41:06.733330 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:06.799091 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 78/155] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Aug 28 00:41:06 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 428 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Aug 28 00:41:06.816943 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19577d25-95ca-4b3b-ace9-d0bd517ccf12 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:06.851580 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 78/156] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:41:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 28 00:41:06.944580 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19577d25-95ca-4b3b-ace9-d0bd517ccf12 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:06.978774 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:06.987649 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-641b7fc7-5fd8-4a98-a65d-89f5473f0a44 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:06.988308 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-641b7fc7-5fd8-4a98-a65d-89f5473f0a44 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:06.988590 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-641b7fc7-5fd8-4a98-a65d-89f5473f0a44 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:06.988865 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-641b7fc7-5fd8-4a98-a65d-89f5473f0a44 None admin] PATH_INFO: `/v3/projects/39243933b7be4190b983169b99d79165` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:06.990064 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-641b7fc7-5fd8-4a98-a65d-89f5473f0a44 None admin] RBAC: Authorizing `identity:get_project(project_id=39243933b7be4190b983169b99d79165)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:06.991550 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-641b7fc7-5fd8-4a98-a65d-89f5473f0a44 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:06.993588 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 79/157] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Aug 28 00:41:06 2022] GET /identity/v3/projects/39243933b7be4190b983169b99d79165 => generated 298 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 28 00:41:06.999737 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de8e3e3d-7f8e-446e-9fbc-9e201d3f7bcc None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:07.139240 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de8e3e3d-7f8e-446e-9fbc-9e201d3f7bcc None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:07.177172 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f38d0edf-e6e1-4896-aa71-110d01cb675a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:07.177791 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f38d0edf-e6e1-4896-aa71-110d01cb675a None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:07.178072 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f38d0edf-e6e1-4896-aa71-110d01cb675a None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:07.178344 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f38d0edf-e6e1-4896-aa71-110d01cb675a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:07.179142 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f38d0edf-e6e1-4896-aa71-110d01cb675a None admin] RBAC: Adding query filter params (role.id=0487ef7e943e4701915cb8ce0f2899d7, scope.project.id=39243933b7be4190b983169b99d79165, user.id=196ac391d55640f99b97d658e79fbe0d) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:07.179442 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f38d0edf-e6e1-4896-aa71-110d01cb675a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:07.180981 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f38d0edf-e6e1-4896-aa71-110d01cb675a None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:07.187366 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 79/158] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Aug 28 00:41:06 2022] GET /identity/v3/role_assignments?user.id=196ac391d55640f99b97d658e79fbe0d&scope.project.id=39243933b7be4190b983169b99d79165&role.id=0487ef7e943e4701915cb8ce0f2899d7 => generated 608 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:07.217653 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-641b7fc7-5fd8-4a98-a65d-89f5473f0a44 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:07.370824 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-641b7fc7-5fd8-4a98-a65d-89f5473f0a44 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:07.415301 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ab58514-ac0a-4aea-b2a1-56a5d8ff1422 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:07.416350 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ab58514-ac0a-4aea-b2a1-56a5d8ff1422 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:07.416932 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ab58514-ac0a-4aea-b2a1-56a5d8ff1422 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:07.417453 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ab58514-ac0a-4aea-b2a1-56a5d8ff1422 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:07.429693 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ab58514-ac0a-4aea-b2a1-56a5d8ff1422 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 28 00:41:07.436035 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-2ab58514-ac0a-4aea-b2a1-56a5d8ff1422 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 28 00:41:07.436035 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:41:07.436035 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:41:07.436035 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:41:07.436035 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:41:07.436035 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:41:07.436035 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:41:07.436035 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:41:07.436035 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:41:07.436035 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:41:07.436035 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:41:07.436035 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:41:07.436035 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 28 00:41:07.436035 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 28 00:41:07.436035 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 28 00:41:07.436035 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 28 00:41:07.436933 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 28 00:41:07.436933 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 28 00:41:07.436933 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 28 00:41:07.436933 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 28 00:41:07.436933 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:41:07.436933 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:41:07.436933 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:41:07.436933 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:41:07.436933 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:41:07.436933 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:41:07.436933 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:41:07.436933 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:41:07.436933 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:41:07.436933 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:41:07.436933 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:41:07.436933 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:41:07.437793 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:41:07.437793 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:41:07.437793 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:41:07.437793 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:41:07.437793 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 28 00:41:07.437793 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 28 00:41:07.437793 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 28 00:41:07.437793 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 28 00:41:07.437793 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 28 00:41:07.437793 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 28 00:41:07.437793 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 28 00:41:07.437793 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:41:07.440500 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 80/159] 10.222.0.42 () {56 vars in 1114 bytes} [Sun Aug 28 00:41:07 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 223 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:41:07.446073 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f38d0edf-e6e1-4896-aa71-110d01cb675a None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:07.449078 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:07.584203 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f38d0edf-e6e1-4896-aa71-110d01cb675a None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:07.623698 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afd57156-fb5d-45e0-889b-3c2294f14461 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:07.624335 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd57156-fb5d-45e0-889b-3c2294f14461 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:07.624618 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd57156-fb5d-45e0-889b-3c2294f14461 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:07.624889 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd57156-fb5d-45e0-889b-3c2294f14461 None admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:07.625682 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afd57156-fb5d-45e0-889b-3c2294f14461 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:07.625992 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afd57156-fb5d-45e0-889b-3c2294f14461 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:07.627410 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afd57156-fb5d-45e0-889b-3c2294f14461 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:07.637897 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 80/160] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Aug 28 00:41:07 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:07.643841 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ab58514-ac0a-4aea-b2a1-56a5d8ff1422 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:07.726767 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:07.778709 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ab58514-ac0a-4aea-b2a1-56a5d8ff1422 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:07.823397 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b6e2bb1-650d-4c4c-be78-badca9f19e88 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:07.824046 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b6e2bb1-650d-4c4c-be78-badca9f19e88 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:07.824322 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b6e2bb1-650d-4c4c-be78-badca9f19e88 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:07.824592 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b6e2bb1-650d-4c4c-be78-badca9f19e88 None admin] PATH_INFO: `/v3/users/196ac391d55640f99b97d658e79fbe0d` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:07.825765 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b6e2bb1-650d-4c4c-be78-badca9f19e88 None admin] RBAC: Authorizing `identity:get_user(user_id=196ac391d55640f99b97d658e79fbe0d)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:07.827241 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b6e2bb1-650d-4c4c-be78-badca9f19e88 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:07.829112 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 81/161] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Aug 28 00:41:07 2022] GET /identity/v3/users/196ac391d55640f99b97d658e79fbe0d => generated 275 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:07.835094 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afd57156-fb5d-45e0-889b-3c2294f14461 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:07.854617 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6d06943-d715-4014-aa5c-53f1905cf07d None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:07.854922 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6d06943-d715-4014-aa5c-53f1905cf07d None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:07.855157 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6d06943-d715-4014-aa5c-53f1905cf07d None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:07.856536 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 82/162] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:41:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 28 00:41:07.864713 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20e88381-33a6-406d-ab7d-a418e73d5a71 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:07.865025 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20e88381-33a6-406d-ab7d-a418e73d5a71 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:07.865282 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20e88381-33a6-406d-ab7d-a418e73d5a71 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:07.898056 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-20e88381-33a6-406d-ab7d-a418e73d5a71 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:41:07.899264 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-20e88381-33a6-406d-ab7d-a418e73d5a71 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:41:07.970330 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:08.045542 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afd57156-fb5d-45e0-889b-3c2294f14461 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:08.071587 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 83/163] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:41:07 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 208 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 28 00:41:08.073612 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60cb6ed1-a069-409d-b701-b5f9b8090abe None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:08.074062 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60cb6ed1-a069-409d-b701-b5f9b8090abe None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:08.077441 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60cb6ed1-a069-409d-b701-b5f9b8090abe None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:08.077441 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 84/164] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:41:08 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:41:08.102943 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-908c32d5-b542-47e0-ae30-ffd24e7d8e72 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:08.103913 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-908c32d5-b542-47e0-ae30-ffd24e7d8e72 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:08.104508 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-908c32d5-b542-47e0-ae30-ffd24e7d8e72 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:08.105010 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-908c32d5-b542-47e0-ae30-ffd24e7d8e72 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:08.108812 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c5d3066-8a04-4a61-a22f-3f23578c5b82 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:08.119009 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c5d3066-8a04-4a61-a22f-3f23578c5b82 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:08.119604 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c5d3066-8a04-4a61-a22f-3f23578c5b82 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:08.162023 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-1c5d3066-8a04-4a61-a22f-3f23578c5b82 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:41:08.163441 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-1c5d3066-8a04-4a61-a22f-3f23578c5b82 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:41:08.203258 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-908c32d5-b542-47e0-ae30-ffd24e7d8e72 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:08.205362 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-908c32d5-b542-47e0-ae30-ffd24e7d8e72 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:08.273815 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:08.306810 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 85/165] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:41:08 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 199 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 28 00:41:08.327048 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:08.379119 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 81/166] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Aug 28 00:41:07 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 545 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 28 00:41:08.384913 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c5d3066-8a04-4a61-a22f-3f23578c5b82 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:08.494241 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c5d3066-8a04-4a61-a22f-3f23578c5b82 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:08.529953 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82f93d60-9239-4515-b69b-7ba06b79d29d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:08.530997 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f93d60-9239-4515-b69b-7ba06b79d29d None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:08.531422 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f93d60-9239-4515-b69b-7ba06b79d29d None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:08.531866 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f93d60-9239-4515-b69b-7ba06b79d29d None admin] PATH_INFO: `/v3/projects/39243933b7be4190b983169b99d79165` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:08.533188 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82f93d60-9239-4515-b69b-7ba06b79d29d None admin] RBAC: Authorizing `identity:get_project(project_id=39243933b7be4190b983169b99d79165)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:08.534812 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82f93d60-9239-4515-b69b-7ba06b79d29d None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:08.536789 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 86/167] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Aug 28 00:41:08 2022] GET /identity/v3/projects/39243933b7be4190b983169b99d79165 => generated 298 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:08.542817 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-908c32d5-b542-47e0-ae30-ffd24e7d8e72 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:08.665303 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-908c32d5-b542-47e0-ae30-ffd24e7d8e72 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:08.697261 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82f93d60-9239-4515-b69b-7ba06b79d29d None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:08.712638 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-629f816a-5faf-4ebc-a576-bc06d054a139 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:08.713791 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-629f816a-5faf-4ebc-a576-bc06d054a139 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:08.714222 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-629f816a-5faf-4ebc-a576-bc06d054a139 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:08.714847 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-629f816a-5faf-4ebc-a576-bc06d054a139 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:08.715965 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-629f816a-5faf-4ebc-a576-bc06d054a139 None admin] RBAC: Adding query filter params (role.id=05caf0846a15452b9a6a79fa59c7fa5a, scope.project.id=39243933b7be4190b983169b99d79165, user.id=196ac391d55640f99b97d658e79fbe0d) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:08.716528 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-629f816a-5faf-4ebc-a576-bc06d054a139 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:08.718426 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-629f816a-5faf-4ebc-a576-bc06d054a139 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:08.727576 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 82/168] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Aug 28 00:41:08 2022] GET /identity/v3/role_assignments?user.id=196ac391d55640f99b97d658e79fbe0d&scope.project.id=39243933b7be4190b983169b99d79165&role.id=05caf0846a15452b9a6a79fa59c7fa5a => generated 608 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:08.804852 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:08.869316 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82f93d60-9239-4515-b69b-7ba06b79d29d None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:08.916738 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0f452b5-6ad9-4a80-8bf7-f1a7aa50516e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:08.917320 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0f452b5-6ad9-4a80-8bf7-f1a7aa50516e None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:08.917597 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0f452b5-6ad9-4a80-8bf7-f1a7aa50516e None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:08.917865 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0f452b5-6ad9-4a80-8bf7-f1a7aa50516e None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:08.936174 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0f452b5-6ad9-4a80-8bf7-f1a7aa50516e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 28 00:41:08.941799 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-e0f452b5-6ad9-4a80-8bf7-f1a7aa50516e None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 28 00:41:08.941799 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:41:08.941799 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:41:08.941799 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:41:08.941799 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:41:08.941799 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:41:08.941799 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:41:08.941799 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:41:08.941799 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:41:08.941799 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:41:08.941799 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:41:08.941799 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:41:08.941799 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 28 00:41:08.941799 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 28 00:41:08.941799 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 28 00:41:08.941799 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 28 00:41:08.942767 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 28 00:41:08.942767 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 28 00:41:08.942767 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 28 00:41:08.942767 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 28 00:41:08.942767 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:41:08.942767 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:41:08.942767 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:41:08.942767 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:41:08.942767 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:41:08.942767 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:41:08.942767 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:41:08.942767 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:41:08.942767 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:41:08.942767 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:41:08.942767 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:41:08.942767 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:41:08.943551 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:41:08.943551 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:41:08.943551 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:41:08.943551 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:41:08.943551 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 28 00:41:08.943551 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 28 00:41:08.943551 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 28 00:41:08.943551 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 28 00:41:08.943551 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 28 00:41:08.943551 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 28 00:41:08.943551 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 28 00:41:08.943551 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:41:08.943551 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 87/169] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Aug 28 00:41:08 2022] GET /identity/v3/roles/admin => generated 83 bytes in 246 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 28 00:41:08.949208 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-629f816a-5faf-4ebc-a576-bc06d054a139 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:08.958700 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0f452b5-6ad9-4a80-8bf7-f1a7aa50516e None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:09.004274 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:09.150772 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0f452b5-6ad9-4a80-8bf7-f1a7aa50516e None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:09.180673 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:09.185045 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-629f816a-5faf-4ebc-a576-bc06d054a139 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:09.223588 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd37c51d-ee9a-447d-b1ad-a5995939fa6f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:09.224628 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd37c51d-ee9a-447d-b1ad-a5995939fa6f None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:09.225142 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd37c51d-ee9a-447d-b1ad-a5995939fa6f None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:09.225605 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd37c51d-ee9a-447d-b1ad-a5995939fa6f None admin] PATH_INFO: `/v3/roles/member` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:09.250020 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd37c51d-ee9a-447d-b1ad-a5995939fa6f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 28 00:41:09.255517 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-bd37c51d-ee9a-447d-b1ad-a5995939fa6f None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 28 00:41:09.255517 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:41:09.255517 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:41:09.255517 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:41:09.255517 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:41:09.255517 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:41:09.255517 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:41:09.255517 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:41:09.255517 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:41:09.255517 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:41:09.255517 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:41:09.255517 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:41:09.255517 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 28 00:41:09.255517 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 28 00:41:09.255517 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 28 00:41:09.255517 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 28 00:41:09.256411 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 28 00:41:09.256411 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 28 00:41:09.256411 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 28 00:41:09.256411 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 28 00:41:09.256411 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:41:09.256411 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:41:09.256411 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:41:09.256411 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:41:09.256411 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:41:09.256411 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:41:09.256411 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:41:09.256411 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:41:09.256411 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:41:09.256411 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:41:09.256411 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:41:09.256411 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:41:09.257221 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:41:09.257221 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:41:09.257221 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:41:09.257221 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:41:09.257221 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 28 00:41:09.257221 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 28 00:41:09.257221 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 28 00:41:09.257221 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 28 00:41:09.257221 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 28 00:41:09.257221 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 28 00:41:09.257221 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 28 00:41:09.257221 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:41:09.267659 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0988081-aacb-4302-a5b5-60bafd2f40eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:09.268280 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0988081-aacb-4302-a5b5-60bafd2f40eb None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:09.268565 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0988081-aacb-4302-a5b5-60bafd2f40eb None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:09.268932 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0988081-aacb-4302-a5b5-60bafd2f40eb None admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:09.269181 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 88/170] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Aug 28 00:41:08 2022] GET /identity/v3/roles/member => generated 84 bytes in 311 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 28 00:41:09.269540 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0988081-aacb-4302-a5b5-60bafd2f40eb None admin] RBAC: Adding query filter params (name=admin) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:09.269840 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0988081-aacb-4302-a5b5-60bafd2f40eb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:09.270725 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd37c51d-ee9a-447d-b1ad-a5995939fa6f None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:09.271380 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0988081-aacb-4302-a5b5-60bafd2f40eb None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:09.291595 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 83/171] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Aug 28 00:41:08 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 343 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:09.298398 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0988081-aacb-4302-a5b5-60bafd2f40eb None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:09.444626 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:09.466783 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd37c51d-ee9a-447d-b1ad-a5995939fa6f None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:09.475401 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0988081-aacb-4302-a5b5-60bafd2f40eb None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:09.557506 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd3c2697-ae28-4d06-95df-f77fc499f5c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:09.558912 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd3c2697-ae28-4d06-95df-f77fc499f5c0 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:09.559690 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd3c2697-ae28-4d06-95df-f77fc499f5c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:09.560658 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd3c2697-ae28-4d06-95df-f77fc499f5c0 None admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:09.561488 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae82341a-4c7c-4c14-855e-48228ec9fa23 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:09.562372 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae82341a-4c7c-4c14-855e-48228ec9fa23 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:09.563839 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd3c2697-ae28-4d06-95df-f77fc499f5c0 None admin] RBAC: Adding query filter params (name=member) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:09.564443 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae82341a-4c7c-4c14-855e-48228ec9fa23 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:09.564986 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae82341a-4c7c-4c14-855e-48228ec9fa23 None admin] PATH_INFO: `/v3/users/e5cb815a43fc4460b6ddf0175f087128` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:09.566357 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd3c2697-ae28-4d06-95df-f77fc499f5c0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:09.568218 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd3c2697-ae28-4d06-95df-f77fc499f5c0 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:09.569176 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae82341a-4c7c-4c14-855e-48228ec9fa23 None admin] RBAC: Authorizing `identity:get_user(user_id=e5cb815a43fc4460b6ddf0175f087128)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:09.571147 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae82341a-4c7c-4c14-855e-48228ec9fa23 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:09.573053 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 84/172] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Aug 28 00:41:09 2022] GET /identity/v3/users/e5cb815a43fc4460b6ddf0175f087128 => generated 247 bytes in 276 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 28 00:41:09.579185 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae82341a-4c7c-4c14-855e-48228ec9fa23 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:09.585373 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 89/173] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Aug 28 00:41:09 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 315 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:09.587478 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb0a015d-7a69-44ec-adaf-90ddd07444ce None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:09.587762 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb0a015d-7a69-44ec-adaf-90ddd07444ce None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:09.588033 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb0a015d-7a69-44ec-adaf-90ddd07444ce None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:09.589584 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 90/174] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:41:09 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:41:09.598821 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a31aba00-34b0-4244-a82e-b79f048012d7 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:09.599135 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a31aba00-34b0-4244-a82e-b79f048012d7 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:09.600225 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a31aba00-34b0-4244-a82e-b79f048012d7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:09.630384 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-a31aba00-34b0-4244-a82e-b79f048012d7 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:41:09.633294 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-a31aba00-34b0-4244-a82e-b79f048012d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:41:09.687387 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:09.762978 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae82341a-4c7c-4c14-855e-48228ec9fa23 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:09.767041 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 91/175] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:41:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 28 00:41:09.767992 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a31aba00-34b0-4244-a82e-b79f048012d7 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:09.830857 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b898f9d-b9ce-4893-8469-46f84273b4fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:09.831691 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b898f9d-b9ce-4893-8469-46f84273b4fd None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:09.831967 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b898f9d-b9ce-4893-8469-46f84273b4fd None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:09.832229 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b898f9d-b9ce-4893-8469-46f84273b4fd None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:09.909368 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:09.984357 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b898f9d-b9ce-4893-8469-46f84273b4fd None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:09.985983 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b898f9d-b9ce-4893-8469-46f84273b4fd None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:10.005001 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a31aba00-34b0-4244-a82e-b79f048012d7 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:10.109875 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4232201-2e20-4553-87f8-a590a815b376 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:10.110457 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4232201-2e20-4553-87f8-a590a815b376 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:10.110741 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4232201-2e20-4553-87f8-a590a815b376 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:10.111007 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4232201-2e20-4553-87f8-a590a815b376 None admin] PATH_INFO: `/v3/users/196ac391d55640f99b97d658e79fbe0d` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:10.112270 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4232201-2e20-4553-87f8-a590a815b376 None admin] RBAC: Authorizing `identity:get_user(user_id=196ac391d55640f99b97d658e79fbe0d)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:10.113778 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4232201-2e20-4553-87f8-a590a815b376 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:10.115584 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 92/176] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Aug 28 00:41:09 2022] GET /identity/v3/users/196ac391d55640f99b97d658e79fbe0d => generated 275 bytes in 348 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:10.116211 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4232201-2e20-4553-87f8-a590a815b376 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:10.148949 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 85/177] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Aug 28 00:41:09 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 570 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 28 00:41:10.150975 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d11fa82d-dd82-4c6e-a345-7a12113d2c9e None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:10.151467 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d11fa82d-dd82-4c6e-a345-7a12113d2c9e None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:10.151955 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d11fa82d-dd82-4c6e-a345-7a12113d2c9e None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:10.153541 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 86/178] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:41:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:41:10.154459 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d11fa82d-dd82-4c6e-a345-7a12113d2c9e None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:10.159375 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:10.238329 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:10.257224 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4232201-2e20-4553-87f8-a590a815b376 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:10.296049 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6268e47-3e65-4939-829f-da6581641a9a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:10.296693 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6268e47-3e65-4939-829f-da6581641a9a None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:10.296996 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6268e47-3e65-4939-829f-da6581641a9a None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:10.297328 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6268e47-3e65-4939-829f-da6581641a9a None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:10.299642 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d11fa82d-dd82-4c6e-a345-7a12113d2c9e None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:10.307810 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6268e47-3e65-4939-829f-da6581641a9a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 28 00:41:10.313243 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-d6268e47-3e65-4939-829f-da6581641a9a None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 28 00:41:10.313243 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:41:10.313243 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:41:10.313243 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:41:10.313243 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:41:10.313243 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:41:10.313243 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:41:10.313243 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:41:10.313243 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:41:10.313243 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:41:10.313243 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:41:10.313243 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:41:10.313243 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 28 00:41:10.313243 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 28 00:41:10.313243 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 28 00:41:10.313243 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 28 00:41:10.314333 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 28 00:41:10.314333 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 28 00:41:10.314333 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 28 00:41:10.314333 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 28 00:41:10.314333 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:41:10.314333 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:41:10.314333 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:41:10.314333 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:41:10.314333 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:41:10.314333 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:41:10.314333 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:41:10.314333 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:41:10.314333 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:41:10.314333 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:41:10.314333 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:41:10.314333 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:41:10.315337 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:41:10.315337 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:41:10.315337 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:41:10.315337 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:41:10.315337 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 28 00:41:10.315337 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 28 00:41:10.315337 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 28 00:41:10.315337 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 28 00:41:10.315337 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 28 00:41:10.315337 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 28 00:41:10.315337 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 28 00:41:10.315337 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:41:10.315337 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 93/179] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Aug 28 00:41:10 2022] GET /identity/v3/roles/admin => generated 83 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:41:10.315337 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6268e47-3e65-4939-829f-da6581641a9a None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:10.343502 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69c517a1-4a40-4ae2-a231-ca4a15031cdb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:10.344178 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69c517a1-4a40-4ae2-a231-ca4a15031cdb None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:10.344455 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69c517a1-4a40-4ae2-a231-ca4a15031cdb None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:10.344724 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69c517a1-4a40-4ae2-a231-ca4a15031cdb None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:10.417108 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:10.417304 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69c517a1-4a40-4ae2-a231-ca4a15031cdb None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:10.418784 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69c517a1-4a40-4ae2-a231-ca4a15031cdb None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:10.428083 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6268e47-3e65-4939-829f-da6581641a9a None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:10.469429 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38d8e4ae-e164-4b2c-aa5e-a1e91d1efd40 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:10.470299 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38d8e4ae-e164-4b2c-aa5e-a1e91d1efd40 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:10.470702 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38d8e4ae-e164-4b2c-aa5e-a1e91d1efd40 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:10.471174 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38d8e4ae-e164-4b2c-aa5e-a1e91d1efd40 None admin] PATH_INFO: `/v3/projects/39243933b7be4190b983169b99d79165` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:10.472797 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38d8e4ae-e164-4b2c-aa5e-a1e91d1efd40 None admin] RBAC: Authorizing `identity:get_project(project_id=39243933b7be4190b983169b99d79165)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:10.474672 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38d8e4ae-e164-4b2c-aa5e-a1e91d1efd40 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:10.476368 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 94/180] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Aug 28 00:41:10 2022] GET /identity/v3/projects/39243933b7be4190b983169b99d79165 => generated 298 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:10.477542 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa24326a-d98f-45f9-b0f9-ed55f1633103 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:10.477822 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa24326a-d98f-45f9-b0f9-ed55f1633103 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:10.478095 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa24326a-d98f-45f9-b0f9-ed55f1633103 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:10.485176 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:10.501556 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-aa24326a-d98f-45f9-b0f9-ed55f1633103 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:41:10.502797 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-aa24326a-d98f-45f9-b0f9-ed55f1633103 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:41:10.531888 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 87/181] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Aug 28 00:41:10 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 378 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 28 00:41:10.533272 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee8d6620-ef15-4469-bc4d-7e672555dc62 None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:10.533563 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee8d6620-ef15-4469-bc4d-7e672555dc62 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:10.533827 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee8d6620-ef15-4469-bc4d-7e672555dc62 None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:10.536997 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 88/182] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:41:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:41:10.536997 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee8d6620-ef15-4469-bc4d-7e672555dc62 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:10.582939 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 95/183] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:41:10 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 28 00:41:10.584233 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b790295-2c51-41c8-b065-36dc236d8871 None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:10.584530 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b790295-2c51-41c8-b065-36dc236d8871 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:10.584793 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b790295-2c51-41c8-b065-36dc236d8871 None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:10.586084 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 96/184] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:41:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:41:10.586786 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b790295-2c51-41c8-b065-36dc236d8871 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:10.655983 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee8d6620-ef15-4469-bc4d-7e672555dc62 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:10.698157 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35e8c8d5-486b-42dc-aa90-a53dfb04ae2c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:10.698789 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e8c8d5-486b-42dc-aa90-a53dfb04ae2c None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:10.699065 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e8c8d5-486b-42dc-aa90-a53dfb04ae2c None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:10.699334 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e8c8d5-486b-42dc-aa90-a53dfb04ae2c None admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:10.700057 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35e8c8d5-486b-42dc-aa90-a53dfb04ae2c None admin] RBAC: Adding query filter params (name=admin) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:10.700362 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35e8c8d5-486b-42dc-aa90-a53dfb04ae2c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:10.701712 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:10.702953 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35e8c8d5-486b-42dc-aa90-a53dfb04ae2c None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:10.713531 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 89/185] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Aug 28 00:41:10 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:10.714172 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b790295-2c51-41c8-b065-36dc236d8871 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:10.714927 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35e8c8d5-486b-42dc-aa90-a53dfb04ae2c None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:10.742464 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:10.753034 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd1f2cbe-7f32-4af0-8d26-88ad185c3ccc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:10.753653 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd1f2cbe-7f32-4af0-8d26-88ad185c3ccc None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:10.753924 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd1f2cbe-7f32-4af0-8d26-88ad185c3ccc None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:10.754191 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd1f2cbe-7f32-4af0-8d26-88ad185c3ccc None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:10.754917 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd1f2cbe-7f32-4af0-8d26-88ad185c3ccc None admin] RBAC: Adding query filter params (role.id=f08d6043261747e3b4763c744eef7dfb, scope.project.id=39243933b7be4190b983169b99d79165, user.id=e5cb815a43fc4460b6ddf0175f087128) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:10.755218 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd1f2cbe-7f32-4af0-8d26-88ad185c3ccc None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:10.756659 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd1f2cbe-7f32-4af0-8d26-88ad185c3ccc None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:10.763598 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 97/186] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Aug 28 00:41:10 2022] GET /identity/v3/role_assignments?user.id=e5cb815a43fc4460b6ddf0175f087128&scope.project.id=39243933b7be4190b983169b99d79165&role.id=f08d6043261747e3b4763c744eef7dfb => generated 608 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:10.766310 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df5452b8-0cb1-46a8-9bb4-76cb0e6f28cb None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:10.766589 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df5452b8-0cb1-46a8-9bb4-76cb0e6f28cb None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:10.766862 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df5452b8-0cb1-46a8-9bb4-76cb0e6f28cb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:10.787569 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-df5452b8-0cb1-46a8-9bb4-76cb0e6f28cb None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:41:10.788824 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-df5452b8-0cb1-46a8-9bb4-76cb0e6f28cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:41:10.862624 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35e8c8d5-486b-42dc-aa90-a53dfb04ae2c None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:10.883448 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 98/187] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:41:10 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 28 00:41:10.891152 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85236dd3-f031-425c-9da3-2316bb1650bb None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:10.892243 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85236dd3-f031-425c-9da3-2316bb1650bb None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:10.893052 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85236dd3-f031-425c-9da3-2316bb1650bb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:10.905555 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5dbe70f-fa95-4742-aef3-17f2cb021f38 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:10.906353 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5dbe70f-fa95-4742-aef3-17f2cb021f38 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:10.906803 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5dbe70f-fa95-4742-aef3-17f2cb021f38 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:10.907182 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5dbe70f-fa95-4742-aef3-17f2cb021f38 None admin] PATH_INFO: `/v3/projects/5341d633667247b9b075a2fcbe4c0156` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:10.908528 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5dbe70f-fa95-4742-aef3-17f2cb021f38 None admin] RBAC: Authorizing `identity:get_project(project_id=5341d633667247b9b075a2fcbe4c0156)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:10.910188 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5dbe70f-fa95-4742-aef3-17f2cb021f38 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:10.912171 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 90/188] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Aug 28 00:41:10 2022] GET /identity/v3/projects/5341d633667247b9b075a2fcbe4c0156 => generated 312 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:10.917156 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5dbe70f-fa95-4742-aef3-17f2cb021f38 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:10.924171 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-85236dd3-f031-425c-9da3-2316bb1650bb None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:41:10.925395 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-85236dd3-f031-425c-9da3-2316bb1650bb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:41:11.017625 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 99/189] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:41:10 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 28 00:41:11.018418 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85236dd3-f031-425c-9da3-2316bb1650bb None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:11.031724 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:11.046431 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:11.052124 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5dbe70f-fa95-4742-aef3-17f2cb021f38 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:11.098284 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f94f7aea-c0a4-4dd1-b95e-6c3696d91017 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:11.099110 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f94f7aea-c0a4-4dd1-b95e-6c3696d91017 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:11.099458 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f94f7aea-c0a4-4dd1-b95e-6c3696d91017 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:11.099802 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f94f7aea-c0a4-4dd1-b95e-6c3696d91017 None admin] PATH_INFO: `/v3/users/6e82746467384e4e9ec43cf94e3a5040` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:11.124496 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f94f7aea-c0a4-4dd1-b95e-6c3696d91017 None admin] RBAC: Authorizing `identity:get_user(user_id=6e82746467384e4e9ec43cf94e3a5040)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:11.126176 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f94f7aea-c0a4-4dd1-b95e-6c3696d91017 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:11.128629 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 91/190] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Aug 28 00:41:10 2022] GET /identity/v3/users/6e82746467384e4e9ec43cf94e3a5040 => generated 283 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:11.129005 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f94f7aea-c0a4-4dd1-b95e-6c3696d91017 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:11.165954 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85236dd3-f031-425c-9da3-2316bb1650bb None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:11.206526 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-301ded25-ceb9-441a-a28f-f0a8b20ed559 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:11.207197 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-301ded25-ceb9-441a-a28f-f0a8b20ed559 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:11.207439 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-301ded25-ceb9-441a-a28f-f0a8b20ed559 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:11.207711 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-301ded25-ceb9-441a-a28f-f0a8b20ed559 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:11.223362 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-301ded25-ceb9-441a-a28f-f0a8b20ed559 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 28 00:41:11.228530 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-301ded25-ceb9-441a-a28f-f0a8b20ed559 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 28 00:41:11.228530 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:41:11.228530 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:41:11.228530 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:41:11.228530 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:41:11.228530 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:41:11.228530 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:41:11.228530 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:41:11.228530 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:41:11.228530 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:41:11.228530 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:41:11.228530 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:41:11.228530 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 28 00:41:11.228530 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 28 00:41:11.228530 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 28 00:41:11.228530 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 28 00:41:11.229491 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 28 00:41:11.229491 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 28 00:41:11.229491 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 28 00:41:11.229491 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 28 00:41:11.229491 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:41:11.229491 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:41:11.229491 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:41:11.229491 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:41:11.229491 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:41:11.229491 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:41:11.229491 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:41:11.229491 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:41:11.229491 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:41:11.229491 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:41:11.229491 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:41:11.229491 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:41:11.230393 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:41:11.230393 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:41:11.230393 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:41:11.230393 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:41:11.230393 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 28 00:41:11.230393 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 28 00:41:11.230393 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 28 00:41:11.230393 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 28 00:41:11.230393 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 28 00:41:11.230393 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 28 00:41:11.230393 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 28 00:41:11.230393 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:41:11.230393 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 100/191] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Aug 28 00:41:11 2022] GET /identity/v3/roles/reader => generated 84 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:41:11.235327 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-301ded25-ceb9-441a-a28f-f0a8b20ed559 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:11.244613 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f94f7aea-c0a4-4dd1-b95e-6c3696d91017 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:11.283402 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a72ddbd-28a9-47f8-ae74-7abc908cf175 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:11.283950 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a72ddbd-28a9-47f8-ae74-7abc908cf175 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:11.284286 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a72ddbd-28a9-47f8-ae74-7abc908cf175 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:11.284514 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a72ddbd-28a9-47f8-ae74-7abc908cf175 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:11.285644 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a72ddbd-28a9-47f8-ae74-7abc908cf175 None admin] RBAC: Adding query filter params (role.id=0487ef7e943e4701915cb8ce0f2899d7, scope.project.id=5341d633667247b9b075a2fcbe4c0156, user.id=196ac391d55640f99b97d658e79fbe0d) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:11.285963 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a72ddbd-28a9-47f8-ae74-7abc908cf175 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:11.287758 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a72ddbd-28a9-47f8-ae74-7abc908cf175 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:11.300995 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 92/192] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Aug 28 00:41:11 2022] GET /identity/v3/role_assignments?user.id=196ac391d55640f99b97d658e79fbe0d&scope.project.id=5341d633667247b9b075a2fcbe4c0156&role.id=0487ef7e943e4701915cb8ce0f2899d7 => generated 608 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:11.300995 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a72ddbd-28a9-47f8-ae74-7abc908cf175 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:11.318998 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:11.334138 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:11.365414 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-301ded25-ceb9-441a-a28f-f0a8b20ed559 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:11.402681 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c690eb22-cd40-4093-a67a-6d1ba7c026ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:11.403321 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c690eb22-cd40-4093-a67a-6d1ba7c026ea None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:11.403582 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c690eb22-cd40-4093-a67a-6d1ba7c026ea None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:11.403850 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c690eb22-cd40-4093-a67a-6d1ba7c026ea None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:11.426993 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a72ddbd-28a9-47f8-ae74-7abc908cf175 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:11.470332 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4ea2954-db18-4e70-b223-4c9e7cabc43d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:11.471515 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4ea2954-db18-4e70-b223-4c9e7cabc43d None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:11.471515 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4ea2954-db18-4e70-b223-4c9e7cabc43d None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:11.471783 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4ea2954-db18-4e70-b223-4c9e7cabc43d None admin] PATH_INFO: `/v3/domains/default` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:11.473918 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4ea2954-db18-4e70-b223-4c9e7cabc43d None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:11.475436 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4ea2954-db18-4e70-b223-4c9e7cabc43d None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:11.477331 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 93/193] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:41:11 2022] GET /identity/v3/domains/default => generated 199 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:11.477915 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4ea2954-db18-4e70-b223-4c9e7cabc43d None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:11.488630 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c690eb22-cd40-4093-a67a-6d1ba7c026ea None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:11.490174 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c690eb22-cd40-4093-a67a-6d1ba7c026ea None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:11.562517 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:11.564059 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:11.584025 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4ea2954-db18-4e70-b223-4c9e7cabc43d None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:11.589322 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 101/194] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Aug 28 00:41:11 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 359 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 28 00:41:11.590162 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c690eb22-cd40-4093-a67a-6d1ba7c026ea None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:11.619816 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e5f3c07-663c-4f45-b7f7-b7722ca031df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:11.620427 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e5f3c07-663c-4f45-b7f7-b7722ca031df None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:11.620700 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e5f3c07-663c-4f45-b7f7-b7722ca031df None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:11.620983 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e5f3c07-663c-4f45-b7f7-b7722ca031df None admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:11.621641 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e5f3c07-663c-4f45-b7f7-b7722ca031df None admin] RBAC: Adding query filter params (name=reader) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:11.621937 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e5f3c07-663c-4f45-b7f7-b7722ca031df None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:11.623314 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e5f3c07-663c-4f45-b7f7-b7722ca031df None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:11.631286 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 94/195] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Aug 28 00:41:11 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:11.632050 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e5f3c07-663c-4f45-b7f7-b7722ca031df None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:11.688040 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c690eb22-cd40-4093-a67a-6d1ba7c026ea None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:11.721837 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7888ce4-2e0a-4800-b586-362f8f61b665 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:11.722463 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7888ce4-2e0a-4800-b586-362f8f61b665 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:11.722739 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7888ce4-2e0a-4800-b586-362f8f61b665 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:11.723005 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7888ce4-2e0a-4800-b586-362f8f61b665 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:11.728807 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e5f3c07-663c-4f45-b7f7-b7722ca031df None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:11.732978 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7888ce4-2e0a-4800-b586-362f8f61b665 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 28 00:41:11.737966 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-d7888ce4-2e0a-4800-b586-362f8f61b665 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 28 00:41:11.737966 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:41:11.737966 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:41:11.737966 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:41:11.737966 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:41:11.737966 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:41:11.737966 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:41:11.737966 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:41:11.737966 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:41:11.737966 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:41:11.737966 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:41:11.737966 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:41:11.737966 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 28 00:41:11.737966 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 28 00:41:11.737966 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 28 00:41:11.737966 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 28 00:41:11.738878 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 28 00:41:11.738878 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 28 00:41:11.738878 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 28 00:41:11.738878 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 28 00:41:11.738878 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:41:11.738878 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:41:11.738878 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:41:11.738878 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:41:11.738878 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:41:11.738878 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:41:11.738878 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:41:11.738878 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:41:11.738878 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:41:11.738878 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:41:11.738878 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:41:11.738878 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:41:11.739857 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:41:11.739857 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:41:11.739857 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:41:11.739857 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:41:11.739857 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 28 00:41:11.739857 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 28 00:41:11.739857 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 28 00:41:11.739857 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 28 00:41:11.739857 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 28 00:41:11.739857 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 28 00:41:11.739857 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 28 00:41:11.739857 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:41:11.739857 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 102/196] 10.222.0.42 () {56 vars in 1114 bytes} [Sun Aug 28 00:41:11 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 149 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:41:11.739857 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7888ce4-2e0a-4800-b586-362f8f61b665 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:11.764806 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a57da50-ecac-478a-87c3-6365c30b22c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:11.765450 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a57da50-ecac-478a-87c3-6365c30b22c7 None admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:11.765728 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a57da50-ecac-478a-87c3-6365c30b22c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:11.766056 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a57da50-ecac-478a-87c3-6365c30b22c7 None admin] PATH_INFO: `/v3/users` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:11.766705 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a57da50-ecac-478a-87c3-6365c30b22c7 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:11.768153 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a57da50-ecac-478a-87c3-6365c30b22c7 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:11.775051 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:11.784874 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:11.801144 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 95/197] 10.222.0.42 () {60 vars in 1132 bytes} [Sun Aug 28 00:41:11 2022] POST /identity/v3/users => generated 297 bytes in 169 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:41:11.802338 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a57da50-ecac-478a-87c3-6365c30b22c7 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:11.851089 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7888ce4-2e0a-4800-b586-362f8f61b665 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:11.894337 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58741b05-351e-4e34-a18f-ceb428e2de62 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:11.894980 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58741b05-351e-4e34-a18f-ceb428e2de62 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:11.895250 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58741b05-351e-4e34-a18f-ceb428e2de62 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:11.895521 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58741b05-351e-4e34-a18f-ceb428e2de62 None admin] PATH_INFO: `/v3/projects/2635535fcc444c74b40bb9c46956c259` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:11.896710 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58741b05-351e-4e34-a18f-ceb428e2de62 None admin] RBAC: Authorizing `identity:get_project(project_id=2635535fcc444c74b40bb9c46956c259)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:11.898228 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58741b05-351e-4e34-a18f-ceb428e2de62 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:11.900077 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 103/198] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Aug 28 00:41:11 2022] GET /identity/v3/projects/2635535fcc444c74b40bb9c46956c259 => generated 302 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:11.900709 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58741b05-351e-4e34-a18f-ceb428e2de62 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:11.924540 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a57da50-ecac-478a-87c3-6365c30b22c7 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:11.967403 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33b2c880-c093-48c4-b652-033c1bc6ae5b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:11.968054 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33b2c880-c093-48c4-b652-033c1bc6ae5b None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:11.968332 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33b2c880-c093-48c4-b652-033c1bc6ae5b None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:11.968602 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33b2c880-c093-48c4-b652-033c1bc6ae5b None admin] PATH_INFO: `/v3/users/b1c830143c754b80a0b378a0ad4b96a5` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:11.970289 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33b2c880-c093-48c4-b652-033c1bc6ae5b None admin] RBAC: Authorizing `identity:get_user(user_id=b1c830143c754b80a0b378a0ad4b96a5)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:11.971783 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33b2c880-c093-48c4-b652-033c1bc6ae5b None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:11.973462 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 96/199] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Aug 28 00:41:11 2022] GET /identity/v3/users/b1c830143c754b80a0b378a0ad4b96a5 => generated 289 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:11.974239 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33b2c880-c093-48c4-b652-033c1bc6ae5b None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:12.022902 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58741b05-351e-4e34-a18f-ceb428e2de62 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:12.024650 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:12.046501 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:12.068755 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1133a321-8f1e-48e8-80b4-ff4ce8466f25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:12.069479 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1133a321-8f1e-48e8-80b4-ff4ce8466f25 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:12.069754 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1133a321-8f1e-48e8-80b4-ff4ce8466f25 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:12.070124 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1133a321-8f1e-48e8-80b4-ff4ce8466f25 None admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:12.070800 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1133a321-8f1e-48e8-80b4-ff4ce8466f25 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:12.071098 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1133a321-8f1e-48e8-80b4-ff4ce8466f25 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:12.072507 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1133a321-8f1e-48e8-80b4-ff4ce8466f25 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:12.084071 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 104/200] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Aug 28 00:41:11 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:12.084986 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1133a321-8f1e-48e8-80b4-ff4ce8466f25 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:12.101267 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33b2c880-c093-48c4-b652-033c1bc6ae5b None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:12.159910 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e0afd47-1531-495e-a446-56074e90e030 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:12.161103 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e0afd47-1531-495e-a446-56074e90e030 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:12.161210 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e0afd47-1531-495e-a446-56074e90e030 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:12.161470 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e0afd47-1531-495e-a446-56074e90e030 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:12.162220 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e0afd47-1531-495e-a446-56074e90e030 None admin] RBAC: Adding query filter params (role.id=f08d6043261747e3b4763c744eef7dfb, scope.project.id=2635535fcc444c74b40bb9c46956c259, user.id=6e82746467384e4e9ec43cf94e3a5040) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:12.162524 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e0afd47-1531-495e-a446-56074e90e030 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:12.163996 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e0afd47-1531-495e-a446-56074e90e030 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:12.171652 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 97/201] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Aug 28 00:41:11 2022] GET /identity/v3/role_assignments?user.id=6e82746467384e4e9ec43cf94e3a5040&scope.project.id=2635535fcc444c74b40bb9c46956c259&role.id=f08d6043261747e3b4763c744eef7dfb => generated 259 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:12.172367 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e0afd47-1531-495e-a446-56074e90e030 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:12.243328 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1133a321-8f1e-48e8-80b4-ff4ce8466f25 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:12.325581 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a11a322-738f-4ed1-8adc-06a80dbb5f7a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:12.328868 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a11a322-738f-4ed1-8adc-06a80dbb5f7a None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:12.329464 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a11a322-738f-4ed1-8adc-06a80dbb5f7a None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:12.330388 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a11a322-738f-4ed1-8adc-06a80dbb5f7a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:12.389594 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e0afd47-1531-495e-a446-56074e90e030 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:12.410451 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:12.448640 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:12.469337 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e1cd3df-8da9-4494-9819-eb4681e62755 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:12.470285 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e1cd3df-8da9-4494-9819-eb4681e62755 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:12.470696 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e1cd3df-8da9-4494-9819-eb4681e62755 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:12.471098 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e1cd3df-8da9-4494-9819-eb4681e62755 None admin] PATH_INFO: `/v3/users/6e82746467384e4e9ec43cf94e3a5040` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:12.475537 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e1cd3df-8da9-4494-9819-eb4681e62755 None admin] RBAC: Authorizing `identity:get_user(user_id=6e82746467384e4e9ec43cf94e3a5040)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:12.480384 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e1cd3df-8da9-4494-9819-eb4681e62755 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:12.483764 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 98/202] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Aug 28 00:41:12 2022] GET /identity/v3/users/6e82746467384e4e9ec43cf94e3a5040 => generated 283 bytes in 311 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:12.492109 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e1cd3df-8da9-4494-9819-eb4681e62755 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:12.555140 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a11a322-738f-4ed1-8adc-06a80dbb5f7a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:12.556674 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a11a322-738f-4ed1-8adc-06a80dbb5f7a None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:12.769240 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e1cd3df-8da9-4494-9819-eb4681e62755 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:12.819175 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cbd49f7-1e99-41c1-920d-3811dfe179f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:12.820175 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cbd49f7-1e99-41c1-920d-3811dfe179f3 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:12.820651 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cbd49f7-1e99-41c1-920d-3811dfe179f3 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:12.821137 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cbd49f7-1e99-41c1-920d-3811dfe179f3 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:12.898119 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:12.907196 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cbd49f7-1e99-41c1-920d-3811dfe179f3 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:12.908973 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cbd49f7-1e99-41c1-920d-3811dfe179f3 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:12.923989 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 105/203] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Aug 28 00:41:12 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 839 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 28 00:41:12.934069 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a11a322-738f-4ed1-8adc-06a80dbb5f7a None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:13.029142 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:13.052818 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 99/204] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Aug 28 00:41:12 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 561 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 28 00:41:13.058693 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cbd49f7-1e99-41c1-920d-3811dfe179f3 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:13.101243 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a11a322-738f-4ed1-8adc-06a80dbb5f7a None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:13.158054 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42821ef2-0f61-4ac9-88d9-95a2088384ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:13.159041 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42821ef2-0f61-4ac9-88d9-95a2088384ce None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:13.159491 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42821ef2-0f61-4ac9-88d9-95a2088384ce None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:13.159944 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42821ef2-0f61-4ac9-88d9-95a2088384ce None admin] PATH_INFO: `/v3/projects/39243933b7be4190b983169b99d79165` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:13.161619 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42821ef2-0f61-4ac9-88d9-95a2088384ce None admin] RBAC: Authorizing `identity:get_project(project_id=39243933b7be4190b983169b99d79165)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:13.163244 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42821ef2-0f61-4ac9-88d9-95a2088384ce None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:13.165120 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 106/205] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Aug 28 00:41:12 2022] GET /identity/v3/projects/39243933b7be4190b983169b99d79165 => generated 298 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:13.171800 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42821ef2-0f61-4ac9-88d9-95a2088384ce None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:13.239291 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cbd49f7-1e99-41c1-920d-3811dfe179f3 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:13.248984 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:13.321202 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20b20c1a-09dd-43b2-bec1-e67875bc2c39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:13.321789 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20b20c1a-09dd-43b2-bec1-e67875bc2c39 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:13.322064 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20b20c1a-09dd-43b2-bec1-e67875bc2c39 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:13.322333 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20b20c1a-09dd-43b2-bec1-e67875bc2c39 None admin] PATH_INFO: `/v3/projects/2635535fcc444c74b40bb9c46956c259` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:13.323648 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20b20c1a-09dd-43b2-bec1-e67875bc2c39 None admin] RBAC: Authorizing `identity:get_project(project_id=2635535fcc444c74b40bb9c46956c259)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:13.325131 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20b20c1a-09dd-43b2-bec1-e67875bc2c39 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:13.330638 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 100/206] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Aug 28 00:41:13 2022] GET /identity/v3/projects/2635535fcc444c74b40bb9c46956c259 => generated 302 bytes in 272 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 28 00:41:13.333738 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20b20c1a-09dd-43b2-bec1-e67875bc2c39 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:13.344448 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42821ef2-0f61-4ac9-88d9-95a2088384ce None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:13.377537 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:13.386867 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89931344-fe6d-41f2-90bc-b4db8e2a8127 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:13.387484 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89931344-fe6d-41f2-90bc-b4db8e2a8127 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:13.387759 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89931344-fe6d-41f2-90bc-b4db8e2a8127 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:13.388027 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89931344-fe6d-41f2-90bc-b4db8e2a8127 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:13.388760 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89931344-fe6d-41f2-90bc-b4db8e2a8127 None admin] RBAC: Adding query filter params (role.id=60823fd2c27d46e98b5d299e86b51918, scope.project.id=39243933b7be4190b983169b99d79165, user.id=b1c830143c754b80a0b378a0ad4b96a5) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:13.389091 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89931344-fe6d-41f2-90bc-b4db8e2a8127 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:13.390528 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89931344-fe6d-41f2-90bc-b4db8e2a8127 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:13.399538 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 107/207] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Aug 28 00:41:13 2022] GET /identity/v3/role_assignments?user.id=b1c830143c754b80a0b378a0ad4b96a5&scope.project.id=39243933b7be4190b983169b99d79165&role.id=60823fd2c27d46e98b5d299e86b51918 => generated 608 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:13.521176 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20b20c1a-09dd-43b2-bec1-e67875bc2c39 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:13.567235 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9bc73ef-16e8-4436-80c4-5386c6e3aee9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:13.567848 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9bc73ef-16e8-4436-80c4-5386c6e3aee9 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:13.568127 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9bc73ef-16e8-4436-80c4-5386c6e3aee9 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:13.568399 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9bc73ef-16e8-4436-80c4-5386c6e3aee9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:13.569168 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9bc73ef-16e8-4436-80c4-5386c6e3aee9 None admin] RBAC: Adding query filter params (role.id=05caf0846a15452b9a6a79fa59c7fa5a, scope.project.id=2635535fcc444c74b40bb9c46956c259, user.id=6e82746467384e4e9ec43cf94e3a5040) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:13.569473 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9bc73ef-16e8-4436-80c4-5386c6e3aee9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:13.570934 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9bc73ef-16e8-4436-80c4-5386c6e3aee9 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:13.576684 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 101/208] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Aug 28 00:41:13 2022] GET /identity/v3/role_assignments?user.id=6e82746467384e4e9ec43cf94e3a5040&scope.project.id=2635535fcc444c74b40bb9c46956c259&role.id=05caf0846a15452b9a6a79fa59c7fa5a => generated 259 bytes in 243 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:14.131178 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c24df2-6b12-4002-813b-a46cd87d5fd0 None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:14.131516 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c24df2-6b12-4002-813b-a46cd87d5fd0 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:14.131723 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c24df2-6b12-4002-813b-a46cd87d5fd0 None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:14.133124 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 108/209] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:41:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:41:14.138671 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43b15c2b-0a85-400a-bd63-9c314cff07f2 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:14.138960 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43b15c2b-0a85-400a-bd63-9c314cff07f2 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:14.139232 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43b15c2b-0a85-400a-bd63-9c314cff07f2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:14.166037 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-43b15c2b-0a85-400a-bd63-9c314cff07f2 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:41:14.167242 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-43b15c2b-0a85-400a-bd63-9c314cff07f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:41:14.191109 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:14.238628 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 102/210] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:41:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 28 00:41:14.283708 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9339962e-403d-4fcc-9bba-5a10e9e32578 None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:14.284323 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9339962e-403d-4fcc-9bba-5a10e9e32578 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:14.284812 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9339962e-403d-4fcc-9bba-5a10e9e32578 None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:14.286426 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 109/211] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:41:14 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 28 00:41:14.292085 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44a070a6-af37-4180-baf9-102edc27c63d None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:14.292597 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44a070a6-af37-4180-baf9-102edc27c63d None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:14.293101 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44a070a6-af37-4180-baf9-102edc27c63d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:14.312137 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-44a070a6-af37-4180-baf9-102edc27c63d None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:41:14.313757 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-44a070a6-af37-4180-baf9-102edc27c63d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:41:14.382495 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 103/212] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:41:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 28 00:41:14.390842 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2f9318c-e2a5-4447-a7f5-dc1903423ea5 None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:14.391718 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2f9318c-e2a5-4447-a7f5-dc1903423ea5 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:14.392206 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2f9318c-e2a5-4447-a7f5-dc1903423ea5 None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:14.393891 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 110/213] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:41:14 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:41:14.399105 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b806a4f-11e0-4ef6-9e15-d0ed732d81e4 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:14.399402 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b806a4f-11e0-4ef6-9e15-d0ed732d81e4 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:14.399682 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b806a4f-11e0-4ef6-9e15-d0ed732d81e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:14.417739 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-0b806a4f-11e0-4ef6-9e15-d0ed732d81e4 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:41:14.418925 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-0b806a4f-11e0-4ef6-9e15-d0ed732d81e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:41:14.487433 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 104/214] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:41:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 28 00:41:14.527147 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2f9318c-e2a5-4447-a7f5-dc1903423ea5 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:14.629303 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2f9318c-e2a5-4447-a7f5-dc1903423ea5 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:14.663995 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f52c5eb-5b5d-418e-96be-b927d233fb2b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:14.665005 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f52c5eb-5b5d-418e-96be-b927d233fb2b None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:14.665538 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f52c5eb-5b5d-418e-96be-b927d233fb2b None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:14.665998 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f52c5eb-5b5d-418e-96be-b927d233fb2b None admin] PATH_INFO: `/v3/roles/member` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:14.676701 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f52c5eb-5b5d-418e-96be-b927d233fb2b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 28 00:41:14.681970 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-8f52c5eb-5b5d-418e-96be-b927d233fb2b None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 28 00:41:14.681970 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:41:14.681970 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:41:14.681970 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:41:14.681970 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:41:14.681970 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:41:14.681970 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:41:14.681970 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:41:14.681970 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:41:14.681970 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:41:14.681970 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:41:14.681970 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:41:14.681970 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 28 00:41:14.681970 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 28 00:41:14.681970 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 28 00:41:14.681970 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 28 00:41:14.682845 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 28 00:41:14.682845 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 28 00:41:14.682845 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 28 00:41:14.682845 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 28 00:41:14.682845 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:41:14.682845 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:41:14.682845 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:41:14.682845 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:41:14.682845 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:41:14.682845 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:41:14.682845 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:41:14.682845 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:41:14.682845 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:41:14.682845 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:41:14.682845 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:41:14.682845 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:41:14.683648 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:41:14.683648 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:41:14.683648 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:41:14.683648 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:41:14.683648 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 28 00:41:14.683648 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 28 00:41:14.683648 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 28 00:41:14.683648 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 28 00:41:14.683648 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 28 00:41:14.683648 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 28 00:41:14.683648 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 28 00:41:14.683648 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:41:14.684277 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b806a4f-11e0-4ef6-9e15-d0ed732d81e4 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:14.686475 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 111/215] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Aug 28 00:41:14 2022] GET /identity/v3/roles/member => generated 84 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:41:14.692463 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f52c5eb-5b5d-418e-96be-b927d233fb2b None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:14.713427 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:14.760692 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:14.805052 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f52c5eb-5b5d-418e-96be-b927d233fb2b None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:14.812694 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b806a4f-11e0-4ef6-9e15-d0ed732d81e4 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:14.844082 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c87a9890-fa9e-4e61-b040-91ff759519b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:14.844964 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c87a9890-fa9e-4e61-b040-91ff759519b2 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:14.845271 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c87a9890-fa9e-4e61-b040-91ff759519b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:14.845549 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c87a9890-fa9e-4e61-b040-91ff759519b2 None admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:14.846234 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c87a9890-fa9e-4e61-b040-91ff759519b2 None admin] RBAC: Adding query filter params (name=member) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:14.846538 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c87a9890-fa9e-4e61-b040-91ff759519b2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:14.847990 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c87a9890-fa9e-4e61-b040-91ff759519b2 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:14.851949 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdef2a9a-753b-4575-954a-c540f6424771 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:14.852930 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdef2a9a-753b-4575-954a-c540f6424771 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:14.853421 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdef2a9a-753b-4575-954a-c540f6424771 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:14.854547 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdef2a9a-753b-4575-954a-c540f6424771 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:14.856290 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdef2a9a-753b-4575-954a-c540f6424771 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:14.857750 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 112/216] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Aug 28 00:41:14 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:14.857750 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c87a9890-fa9e-4e61-b040-91ff759519b2 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:14.862983 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdef2a9a-753b-4575-954a-c540f6424771 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:14.864113 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 105/217] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:41:14 2022] GET /identity/v3/domains/default => generated 199 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:14.865884 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdef2a9a-753b-4575-954a-c540f6424771 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:14.965513 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:14.980603 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdef2a9a-753b-4575-954a-c540f6424771 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:14.983540 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c87a9890-fa9e-4e61-b040-91ff759519b2 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:15.001478 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:15.016465 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9397e35a-d630-4e0a-b789-ce5918da39e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:15.017385 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9397e35a-d630-4e0a-b789-ce5918da39e7 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:15.017666 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9397e35a-d630-4e0a-b789-ce5918da39e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:15.017936 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9397e35a-d630-4e0a-b789-ce5918da39e7 None admin] PATH_INFO: `/v3/users/cbabe7b652f54aa096188a7ed46cfeb0` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:15.020535 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bdeecd8-e797-4986-a7e0-aaa6f0092978 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:15.021313 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bdeecd8-e797-4986-a7e0-aaa6f0092978 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:15.021692 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bdeecd8-e797-4986-a7e0-aaa6f0092978 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:15.022061 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bdeecd8-e797-4986-a7e0-aaa6f0092978 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:15.031958 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9397e35a-d630-4e0a-b789-ce5918da39e7 None admin] RBAC: Authorizing `identity:get_user(user_id=cbabe7b652f54aa096188a7ed46cfeb0)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:15.032949 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bdeecd8-e797-4986-a7e0-aaa6f0092978 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 28 00:41:15.033469 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9397e35a-d630-4e0a-b789-ce5918da39e7 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:15.036319 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 106/218] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Aug 28 00:41:14 2022] GET /identity/v3/users/cbabe7b652f54aa096188a7ed46cfeb0 => generated 297 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:15.036319 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9397e35a-d630-4e0a-b789-ce5918da39e7 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:15.042668 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-2bdeecd8-e797-4986-a7e0-aaa6f0092978 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 28 00:41:15.042668 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:41:15.042668 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:41:15.042668 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:41:15.042668 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:41:15.042668 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:41:15.042668 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:41:15.042668 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:41:15.042668 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:41:15.042668 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:41:15.042668 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:41:15.042668 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:41:15.042668 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 28 00:41:15.042668 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 28 00:41:15.042668 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 28 00:41:15.042668 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 28 00:41:15.043777 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 28 00:41:15.043777 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 28 00:41:15.043777 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 28 00:41:15.043777 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 28 00:41:15.043777 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:41:15.043777 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:41:15.043777 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:41:15.043777 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:41:15.043777 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:41:15.043777 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:41:15.043777 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:41:15.043777 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:41:15.043777 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:41:15.043777 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:41:15.043777 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:41:15.043777 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:41:15.044754 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:41:15.044754 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:41:15.044754 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:41:15.044754 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:41:15.044754 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 28 00:41:15.044754 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 28 00:41:15.044754 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 28 00:41:15.044754 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 28 00:41:15.044754 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 28 00:41:15.044754 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 28 00:41:15.044754 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 28 00:41:15.044754 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:41:15.046512 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 113/219] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Aug 28 00:41:14 2022] GET /identity/v3/roles/admin => generated 83 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:41:15.047304 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bdeecd8-e797-4986-a7e0-aaa6f0092978 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:15.157718 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9397e35a-d630-4e0a-b789-ce5918da39e7 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:15.166732 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bdeecd8-e797-4986-a7e0-aaa6f0092978 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:15.198785 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b67fe427-c066-4c0c-b3e3-d5796cabe0ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:15.199653 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b67fe427-c066-4c0c-b3e3-d5796cabe0ad None admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:15.200066 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b67fe427-c066-4c0c-b3e3-d5796cabe0ad None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:15.200505 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b67fe427-c066-4c0c-b3e3-d5796cabe0ad None admin] PATH_INFO: `/v3/users` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:15.201330 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b67fe427-c066-4c0c-b3e3-d5796cabe0ad None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:15.202909 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29220a11-e7ec-446b-9b56-0136d68427c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:15.203113 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b67fe427-c066-4c0c-b3e3-d5796cabe0ad None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:15.203577 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29220a11-e7ec-446b-9b56-0136d68427c8 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:15.203856 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29220a11-e7ec-446b-9b56-0136d68427c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:15.204126 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29220a11-e7ec-446b-9b56-0136d68427c8 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:15.228862 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:15.245744 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 107/220] 10.222.0.42 () {60 vars in 1132 bytes} [Sun Aug 28 00:41:15 2022] POST /identity/v3/users => generated 297 bytes in 210 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:41:15.247575 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b67fe427-c066-4c0c-b3e3-d5796cabe0ad None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:15.269944 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:15.297290 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29220a11-e7ec-446b-9b56-0136d68427c8 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:15.298805 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29220a11-e7ec-446b-9b56-0136d68427c8 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:15.398184 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b67fe427-c066-4c0c-b3e3-d5796cabe0ad None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:15.413123 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 114/221] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Aug 28 00:41:15 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 366 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 28 00:41:15.418696 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29220a11-e7ec-446b-9b56-0136d68427c8 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:15.448994 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ff07776-1e71-4f77-81c1-f3f90b7c8a44 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:15.449703 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ff07776-1e71-4f77-81c1-f3f90b7c8a44 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:15.449981 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ff07776-1e71-4f77-81c1-f3f90b7c8a44 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:15.450246 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ff07776-1e71-4f77-81c1-f3f90b7c8a44 None admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:15.450918 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ff07776-1e71-4f77-81c1-f3f90b7c8a44 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:15.451215 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ff07776-1e71-4f77-81c1-f3f90b7c8a44 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:15.452820 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ff07776-1e71-4f77-81c1-f3f90b7c8a44 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:15.462299 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 108/222] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Aug 28 00:41:15 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:15.467845 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ff07776-1e71-4f77-81c1-f3f90b7c8a44 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:15.518783 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:15.552018 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:15.557814 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29220a11-e7ec-446b-9b56-0136d68427c8 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:15.601137 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ff07776-1e71-4f77-81c1-f3f90b7c8a44 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:15.606507 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5becd6f1-88a2-496a-bf05-c19d116b2671 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:15.607156 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5becd6f1-88a2-496a-bf05-c19d116b2671 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:15.607434 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5becd6f1-88a2-496a-bf05-c19d116b2671 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:15.607704 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5becd6f1-88a2-496a-bf05-c19d116b2671 None admin] PATH_INFO: `/v3/projects/2635535fcc444c74b40bb9c46956c259` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:15.609598 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5becd6f1-88a2-496a-bf05-c19d116b2671 None admin] RBAC: Authorizing `identity:get_project(project_id=2635535fcc444c74b40bb9c46956c259)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:15.611698 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5becd6f1-88a2-496a-bf05-c19d116b2671 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:15.613829 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 115/223] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Aug 28 00:41:15 2022] GET /identity/v3/projects/2635535fcc444c74b40bb9c46956c259 => generated 302 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 28 00:41:15.615379 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac06a44e-7e1a-4340-9792-a9cb229c0a07 None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:15.615663 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac06a44e-7e1a-4340-9792-a9cb229c0a07 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:15.616083 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac06a44e-7e1a-4340-9792-a9cb229c0a07 None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:15.617511 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 116/224] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:41:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:41:15.622791 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac06a44e-7e1a-4340-9792-a9cb229c0a07 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:15.649166 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2a2d0f9-441f-418b-9d0d-2ba9b106daa9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:15.649902 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2a2d0f9-441f-418b-9d0d-2ba9b106daa9 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:15.650256 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2a2d0f9-441f-418b-9d0d-2ba9b106daa9 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:15.650534 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2a2d0f9-441f-418b-9d0d-2ba9b106daa9 None admin] PATH_INFO: `/v3/users/6e82746467384e4e9ec43cf94e3a5040` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:15.651710 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2a2d0f9-441f-418b-9d0d-2ba9b106daa9 None admin] RBAC: Authorizing `identity:get_user(user_id=6e82746467384e4e9ec43cf94e3a5040)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:15.653351 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2a2d0f9-441f-418b-9d0d-2ba9b106daa9 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:15.655012 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 109/225] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Aug 28 00:41:15 2022] GET /identity/v3/users/6e82746467384e4e9ec43cf94e3a5040 => generated 283 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 28 00:41:15.656175 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5af33171-b50a-4f78-9609-408017c6264c None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:15.656491 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5af33171-b50a-4f78-9609-408017c6264c None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:15.656771 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5af33171-b50a-4f78-9609-408017c6264c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:15.676533 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-5af33171-b50a-4f78-9609-408017c6264c None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:41:15.678235 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-5af33171-b50a-4f78-9609-408017c6264c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:41:15.741228 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac06a44e-7e1a-4340-9792-a9cb229c0a07 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:15.770063 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 110/226] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:41:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 28 00:41:15.771143 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5af33171-b50a-4f78-9609-408017c6264c None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:15.782420 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bae54bd5-db68-4694-9c60-ab8bc42732b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:15.783099 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bae54bd5-db68-4694-9c60-ab8bc42732b9 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:15.783418 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bae54bd5-db68-4694-9c60-ab8bc42732b9 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:15.783690 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bae54bd5-db68-4694-9c60-ab8bc42732b9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:15.784472 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bae54bd5-db68-4694-9c60-ab8bc42732b9 None admin] RBAC: Adding query filter params (role.id=0487ef7e943e4701915cb8ce0f2899d7, scope.project.id=2635535fcc444c74b40bb9c46956c259, user.id=cbabe7b652f54aa096188a7ed46cfeb0) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:15.784833 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bae54bd5-db68-4694-9c60-ab8bc42732b9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:15.786457 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bae54bd5-db68-4694-9c60-ab8bc42732b9 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:15.792990 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 117/227] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Aug 28 00:41:15 2022] GET /identity/v3/role_assignments?user.id=cbabe7b652f54aa096188a7ed46cfeb0&scope.project.id=2635535fcc444c74b40bb9c46956c259&role.id=0487ef7e943e4701915cb8ce0f2899d7 => generated 259 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 28 00:41:15.878740 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:15.924092 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5af33171-b50a-4f78-9609-408017c6264c None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:15.960996 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87a190a7-0ed3-448b-ad6d-ccd66ad50769 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:15.961642 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a190a7-0ed3-448b-ad6d-ccd66ad50769 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:15.961918 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a190a7-0ed3-448b-ad6d-ccd66ad50769 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:15.962186 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a190a7-0ed3-448b-ad6d-ccd66ad50769 None admin] PATH_INFO: `/v3/projects/2635535fcc444c74b40bb9c46956c259` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:15.963317 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87a190a7-0ed3-448b-ad6d-ccd66ad50769 None admin] RBAC: Authorizing `identity:get_project(project_id=2635535fcc444c74b40bb9c46956c259)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:15.964761 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87a190a7-0ed3-448b-ad6d-ccd66ad50769 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:15.966939 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 111/228] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Aug 28 00:41:15 2022] GET /identity/v3/projects/2635535fcc444c74b40bb9c46956c259 => generated 302 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:15.972628 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bae54bd5-db68-4694-9c60-ab8bc42732b9 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:15.986661 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:16.093297 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bae54bd5-db68-4694-9c60-ab8bc42732b9 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:16.138755 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc3e0f83-0d6d-4c0d-8bc1-72f8aeb8c543 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:16.139803 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc3e0f83-0d6d-4c0d-8bc1-72f8aeb8c543 None admin] REQUEST_METHOD: `PUT` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:16.140290 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc3e0f83-0d6d-4c0d-8bc1-72f8aeb8c543 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:16.140841 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc3e0f83-0d6d-4c0d-8bc1-72f8aeb8c543 None admin] PATH_INFO: `/v3/projects/2635535fcc444c74b40bb9c46956c259/users/6e82746467384e4e9ec43cf94e3a5040/roles/f08d6043261747e3b4763c744eef7dfb` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:16.143013 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc3e0f83-0d6d-4c0d-8bc1-72f8aeb8c543 None admin] RBAC: Authorizing `identity:create_grant(project_id=2635535fcc444c74b40bb9c46956c259, user_id=6e82746467384e4e9ec43cf94e3a5040, role_id=f08d6043261747e3b4763c744eef7dfb)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:16.144699 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc3e0f83-0d6d-4c0d-8bc1-72f8aeb8c543 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:16.161247 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 118/229] 10.222.0.42 () {58 vars in 1439 bytes} [Sun Aug 28 00:41:15 2022] PUT /identity/v3/projects/2635535fcc444c74b40bb9c46956c259/users/6e82746467384e4e9ec43cf94e3a5040/roles/f08d6043261747e3b4763c744eef7dfb => generated 0 bytes in 189 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:41:16.170831 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87a190a7-0ed3-448b-ad6d-ccd66ad50769 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:16.292189 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87a190a7-0ed3-448b-ad6d-ccd66ad50769 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:16.327764 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:16.345000 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-848d6022-6f83-45dd-b665-ce91e11724a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:16.345629 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-848d6022-6f83-45dd-b665-ce91e11724a8 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:16.345907 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-848d6022-6f83-45dd-b665-ce91e11724a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:16.346177 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-848d6022-6f83-45dd-b665-ce91e11724a8 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:16.363205 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-848d6022-6f83-45dd-b665-ce91e11724a8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 28 00:41:16.368162 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-848d6022-6f83-45dd-b665-ce91e11724a8 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 28 00:41:16.368162 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:41:16.368162 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:41:16.368162 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:41:16.368162 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:41:16.368162 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:41:16.368162 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:41:16.368162 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:41:16.368162 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:41:16.368162 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:41:16.368162 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:41:16.368162 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:41:16.368162 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 28 00:41:16.368162 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 28 00:41:16.368162 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 28 00:41:16.368162 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 28 00:41:16.369072 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 28 00:41:16.369072 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 28 00:41:16.369072 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 28 00:41:16.369072 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 28 00:41:16.369072 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:41:16.369072 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:41:16.369072 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:41:16.369072 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:41:16.369072 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:41:16.369072 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:41:16.369072 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:41:16.369072 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:41:16.369072 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:41:16.369072 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:41:16.369072 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:41:16.369072 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:41:16.370350 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:41:16.370350 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:41:16.370350 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:41:16.370350 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:41:16.370350 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 28 00:41:16.370350 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 28 00:41:16.370350 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 28 00:41:16.370350 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 28 00:41:16.370350 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 28 00:41:16.370350 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 28 00:41:16.370350 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 28 00:41:16.370350 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:41:16.370350 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 112/230] 10.222.0.42 () {56 vars in 1114 bytes} [Sun Aug 28 00:41:16 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:41:16.377218 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc3e0f83-0d6d-4c0d-8bc1-72f8aeb8c543 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:16.458235 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:16.495858 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc3e0f83-0d6d-4c0d-8bc1-72f8aeb8c543 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:16.534761 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9597799-f1d1-4b4e-afc4-b50be9f2e381 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:16.535632 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9597799-f1d1-4b4e-afc4-b50be9f2e381 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:16.536075 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9597799-f1d1-4b4e-afc4-b50be9f2e381 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:16.536490 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9597799-f1d1-4b4e-afc4-b50be9f2e381 None admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:16.537327 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9597799-f1d1-4b4e-afc4-b50be9f2e381 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:16.537795 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9597799-f1d1-4b4e-afc4-b50be9f2e381 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:16.539330 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9597799-f1d1-4b4e-afc4-b50be9f2e381 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:16.548040 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 119/231] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Aug 28 00:41:16 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:16.553824 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-848d6022-6f83-45dd-b665-ce91e11724a8 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:16.708788 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-848d6022-6f83-45dd-b665-ce91e11724a8 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:16.760522 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d7ef757-831f-4de6-85d1-40b7c5db584c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:16.761213 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d7ef757-831f-4de6-85d1-40b7c5db584c None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:16.761539 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d7ef757-831f-4de6-85d1-40b7c5db584c None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:16.761820 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d7ef757-831f-4de6-85d1-40b7c5db584c None admin] PATH_INFO: `/v3/users/6e82746467384e4e9ec43cf94e3a5040` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:16.762994 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d7ef757-831f-4de6-85d1-40b7c5db584c None admin] RBAC: Authorizing `identity:get_user(user_id=6e82746467384e4e9ec43cf94e3a5040)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:16.764482 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d7ef757-831f-4de6-85d1-40b7c5db584c None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:16.766490 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 113/232] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Aug 28 00:41:16 2022] GET /identity/v3/users/6e82746467384e4e9ec43cf94e3a5040 => generated 283 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:16.771712 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9597799-f1d1-4b4e-afc4-b50be9f2e381 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:16.915479 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9597799-f1d1-4b4e-afc4-b50be9f2e381 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:16.950187 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:16.958130 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-554e7486-9c12-496b-9259-392667cd445f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:16.959374 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-554e7486-9c12-496b-9259-392667cd445f None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:16.959585 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-554e7486-9c12-496b-9259-392667cd445f None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:16.959867 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-554e7486-9c12-496b-9259-392667cd445f None admin] PATH_INFO: `/v3/projects/2635535fcc444c74b40bb9c46956c259` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:16.961072 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-554e7486-9c12-496b-9259-392667cd445f None admin] RBAC: Authorizing `identity:get_project(project_id=2635535fcc444c74b40bb9c46956c259)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:16.962603 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-554e7486-9c12-496b-9259-392667cd445f None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:16.964743 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 120/233] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Aug 28 00:41:16 2022] GET /identity/v3/projects/2635535fcc444c74b40bb9c46956c259 => generated 302 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:16.973939 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d7ef757-831f-4de6-85d1-40b7c5db584c None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:17.065593 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:17.137943 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d7ef757-831f-4de6-85d1-40b7c5db584c None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:17.186833 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f439357b-e05e-4ba2-9a47-96b834d6ab1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:17.187509 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f439357b-e05e-4ba2-9a47-96b834d6ab1b None admin] REQUEST_METHOD: `PUT` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:17.187856 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f439357b-e05e-4ba2-9a47-96b834d6ab1b None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:17.188133 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f439357b-e05e-4ba2-9a47-96b834d6ab1b None admin] PATH_INFO: `/v3/projects/2635535fcc444c74b40bb9c46956c259/users/6e82746467384e4e9ec43cf94e3a5040/roles/05caf0846a15452b9a6a79fa59c7fa5a` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:17.194469 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f439357b-e05e-4ba2-9a47-96b834d6ab1b None admin] RBAC: Authorizing `identity:create_grant(project_id=2635535fcc444c74b40bb9c46956c259, user_id=6e82746467384e4e9ec43cf94e3a5040, role_id=05caf0846a15452b9a6a79fa59c7fa5a)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:17.195960 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f439357b-e05e-4ba2-9a47-96b834d6ab1b None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:17.222619 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 114/234] 10.222.0.42 () {58 vars in 1439 bytes} [Sun Aug 28 00:41:16 2022] PUT /identity/v3/projects/2635535fcc444c74b40bb9c46956c259/users/6e82746467384e4e9ec43cf94e3a5040/roles/05caf0846a15452b9a6a79fa59c7fa5a => generated 0 bytes in 249 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:41:17.731102 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2fb56a6-9eb1-45fc-987d-62adc269cc9d None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:17.731772 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2fb56a6-9eb1-45fc-987d-62adc269cc9d None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:17.731772 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2fb56a6-9eb1-45fc-987d-62adc269cc9d None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:17.733414 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 121/235] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:41:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:41:17.750983 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4795654-3c98-4fda-ba83-5056742e152b None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:17.751267 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4795654-3c98-4fda-ba83-5056742e152b None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:17.751527 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4795654-3c98-4fda-ba83-5056742e152b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:17.774283 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-e4795654-3c98-4fda-ba83-5056742e152b None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:41:17.775518 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-e4795654-3c98-4fda-ba83-5056742e152b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:41:17.820928 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97275f24-104d-4ade-87cc-5e70e58dd1cc None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:17.821604 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97275f24-104d-4ade-87cc-5e70e58dd1cc None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:17.822101 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97275f24-104d-4ade-87cc-5e70e58dd1cc None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:17.823672 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 122/236] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:41:17 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 28 00:41:17.829242 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3193cfb-c2c6-4a1c-abeb-fdb7fc5390f0 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:17.830134 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3193cfb-c2c6-4a1c-abeb-fdb7fc5390f0 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:17.830609 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3193cfb-c2c6-4a1c-abeb-fdb7fc5390f0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:17.849076 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-e3193cfb-c2c6-4a1c-abeb-fdb7fc5390f0 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:41:17.850663 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-e3193cfb-c2c6-4a1c-abeb-fdb7fc5390f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:41:17.868386 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 115/237] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:41:17 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 28 00:41:17.949093 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 123/238] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:41:17 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 28 00:41:18.159015 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-064acadb-e3ab-42b1-9b98-d9cb72f7f173 None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:18.159524 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-064acadb-e3ab-42b1-9b98-d9cb72f7f173 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:18.159629 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-064acadb-e3ab-42b1-9b98-d9cb72f7f173 None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:18.161210 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 116/239] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:41:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:41:18.166500 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-492cf248-145b-4eb3-b129-8b7210201efa None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:18.166832 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-492cf248-145b-4eb3-b129-8b7210201efa None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:18.167104 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-492cf248-145b-4eb3-b129-8b7210201efa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:18.185421 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-492cf248-145b-4eb3-b129-8b7210201efa None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:41:18.186599 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-492cf248-145b-4eb3-b129-8b7210201efa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:41:18.264321 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-064acadb-e3ab-42b1-9b98-d9cb72f7f173 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:18.283341 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 124/240] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:41:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 28 00:41:18.284322 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-492cf248-145b-4eb3-b129-8b7210201efa None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:18.345353 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:18.400847 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-064acadb-e3ab-42b1-9b98-d9cb72f7f173 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:18.431736 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:18.443057 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fe11e5f-665b-4aab-8433-0bb4bb57d6e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:18.443707 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe11e5f-665b-4aab-8433-0bb4bb57d6e4 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:18.443990 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe11e5f-665b-4aab-8433-0bb4bb57d6e4 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:18.444267 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe11e5f-665b-4aab-8433-0bb4bb57d6e4 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:18.446450 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fe11e5f-665b-4aab-8433-0bb4bb57d6e4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:18.447928 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fe11e5f-665b-4aab-8433-0bb4bb57d6e4 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:18.449566 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 117/241] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:41:18 2022] GET /identity/v3/domains/default => generated 199 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:18.456627 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fe11e5f-665b-4aab-8433-0bb4bb57d6e4 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:18.516938 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-492cf248-145b-4eb3-b129-8b7210201efa None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:18.563404 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a86def5-3b05-4010-957d-190791ef82b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:18.564042 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a86def5-3b05-4010-957d-190791ef82b4 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:18.564322 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a86def5-3b05-4010-957d-190791ef82b4 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:18.564595 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a86def5-3b05-4010-957d-190791ef82b4 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:18.587155 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a86def5-3b05-4010-957d-190791ef82b4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 28 00:41:18.592150 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-9a86def5-3b05-4010-957d-190791ef82b4 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 28 00:41:18.592150 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:41:18.592150 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:41:18.592150 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:41:18.592150 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:41:18.592150 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:41:18.592150 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:41:18.592150 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:41:18.592150 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:41:18.592150 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:41:18.592150 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:41:18.592150 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:41:18.592150 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 28 00:41:18.592150 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 28 00:41:18.592150 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 28 00:41:18.592150 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 28 00:41:18.593051 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 28 00:41:18.593051 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 28 00:41:18.593051 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 28 00:41:18.593051 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 28 00:41:18.593051 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:41:18.593051 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:41:18.593051 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:41:18.593051 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:41:18.593051 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:41:18.593051 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:41:18.593051 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:41:18.593051 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:41:18.593051 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:41:18.593051 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:41:18.593051 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:41:18.593051 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:41:18.593938 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:41:18.593938 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:41:18.593938 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:41:18.593938 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:41:18.593938 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 28 00:41:18.593938 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 28 00:41:18.593938 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 28 00:41:18.593938 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 28 00:41:18.593938 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 28 00:41:18.593938 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 28 00:41:18.593938 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 28 00:41:18.593938 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:41:18.593938 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 125/242] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Aug 28 00:41:18 2022] GET /identity/v3/roles/reader => generated 84 bytes in 310 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:41:18.601371 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a86def5-3b05-4010-957d-190791ef82b4 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:18.637579 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fe11e5f-665b-4aab-8433-0bb4bb57d6e4 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:18.660619 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:18.698167 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4d801cf-f202-428b-bffb-17dc11a46ce1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:18.698843 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4d801cf-f202-428b-bffb-17dc11a46ce1 None admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:18.699127 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4d801cf-f202-428b-bffb-17dc11a46ce1 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:18.699397 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4d801cf-f202-428b-bffb-17dc11a46ce1 None admin] PATH_INFO: `/v3/users` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:18.700613 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4d801cf-f202-428b-bffb-17dc11a46ce1 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:18.702402 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4d801cf-f202-428b-bffb-17dc11a46ce1 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:18.713465 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a86def5-3b05-4010-957d-190791ef82b4 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:18.731772 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 118/243] 10.222.0.42 () {60 vars in 1132 bytes} [Sun Aug 28 00:41:18 2022] POST /identity/v3/users => generated 293 bytes in 275 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:41:18.732706 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4d801cf-f202-428b-bffb-17dc11a46ce1 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:18.753532 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:18.785345 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef862ca4-91c3-4bae-93e6-7525b6570dc8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:18.786002 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef862ca4-91c3-4bae-93e6-7525b6570dc8 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:18.786276 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef862ca4-91c3-4bae-93e6-7525b6570dc8 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:18.786543 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef862ca4-91c3-4bae-93e6-7525b6570dc8 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:18.805322 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef862ca4-91c3-4bae-93e6-7525b6570dc8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 28 00:41:18.810565 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-ef862ca4-91c3-4bae-93e6-7525b6570dc8 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 28 00:41:18.810565 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:41:18.810565 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:41:18.810565 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:41:18.810565 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:41:18.810565 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:41:18.810565 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:41:18.810565 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:41:18.810565 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:41:18.810565 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:41:18.810565 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:41:18.810565 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:41:18.810565 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 28 00:41:18.810565 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 28 00:41:18.810565 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 28 00:41:18.810565 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 28 00:41:18.812802 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 28 00:41:18.812802 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 28 00:41:18.812802 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 28 00:41:18.812802 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 28 00:41:18.812802 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:41:18.812802 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:41:18.812802 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:41:18.812802 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:41:18.812802 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:41:18.812802 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:41:18.812802 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:41:18.812802 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:41:18.812802 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:41:18.812802 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:41:18.812802 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:41:18.812802 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:41:18.814752 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:41:18.814752 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:41:18.814752 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:41:18.814752 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:41:18.814752 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 28 00:41:18.814752 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 28 00:41:18.814752 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 28 00:41:18.814752 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 28 00:41:18.814752 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 28 00:41:18.814752 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 28 00:41:18.814752 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 28 00:41:18.814752 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:41:18.814752 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 126/244] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Aug 28 00:41:18 2022] GET /identity/v3/roles/member => generated 84 bytes in 218 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:41:18.821041 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d516b1e-eb16-4648-ada5-b8cb687e09d6 None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:18.821041 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d516b1e-eb16-4648-ada5-b8cb687e09d6 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:18.821041 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d516b1e-eb16-4648-ada5-b8cb687e09d6 None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:18.821041 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 127/245] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:41:18 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:41:18.825785 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d516b1e-eb16-4648-ada5-b8cb687e09d6 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:18.894236 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4d801cf-f202-428b-bffb-17dc11a46ce1 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:18.930636 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-951b5920-0454-4dd2-9468-7ef49990c72b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:18.931655 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-951b5920-0454-4dd2-9468-7ef49990c72b None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:18.932173 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-951b5920-0454-4dd2-9468-7ef49990c72b None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:18.932627 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-951b5920-0454-4dd2-9468-7ef49990c72b None admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:18.934490 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-951b5920-0454-4dd2-9468-7ef49990c72b None admin] RBAC: Adding query filter params (name=reader) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:18.934803 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-951b5920-0454-4dd2-9468-7ef49990c72b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:18.936217 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-951b5920-0454-4dd2-9468-7ef49990c72b None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:18.938685 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d516b1e-eb16-4648-ada5-b8cb687e09d6 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:18.946292 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 119/246] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Aug 28 00:41:18 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:18.950269 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e941d1f9-da0f-486c-8c59-827da95caf50 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:18.950560 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e941d1f9-da0f-486c-8c59-827da95caf50 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:18.951063 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e941d1f9-da0f-486c-8c59-827da95caf50 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:18.973068 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-e941d1f9-da0f-486c-8c59-827da95caf50 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:41:18.974392 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-e941d1f9-da0f-486c-8c59-827da95caf50 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:41:18.980732 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e7d9b3b-cb46-467f-90de-4233c77409cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:18.981786 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e7d9b3b-cb46-467f-90de-4233c77409cb None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:18.982199 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e7d9b3b-cb46-467f-90de-4233c77409cb None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:18.982598 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e7d9b3b-cb46-467f-90de-4233c77409cb None admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:18.983987 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e7d9b3b-cb46-467f-90de-4233c77409cb None admin] RBAC: Adding query filter params (name=member) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:18.984513 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e7d9b3b-cb46-467f-90de-4233c77409cb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:18.986737 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e7d9b3b-cb46-467f-90de-4233c77409cb None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:18.996050 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 128/247] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Aug 28 00:41:18 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:18.997128 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e7d9b3b-cb46-467f-90de-4233c77409cb None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:19.012024 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:19.063801 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:19.070663 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 120/248] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:41:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 28 00:41:19.071371 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e941d1f9-da0f-486c-8c59-827da95caf50 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:19.170930 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e7d9b3b-cb46-467f-90de-4233c77409cb None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:19.212108 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e941d1f9-da0f-486c-8c59-827da95caf50 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:19.224392 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c3e82c5-733f-436f-9dd4-46f282a4117f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:19.225667 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c3e82c5-733f-436f-9dd4-46f282a4117f None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:19.226019 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c3e82c5-733f-436f-9dd4-46f282a4117f None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:19.226362 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c3e82c5-733f-436f-9dd4-46f282a4117f None admin] PATH_INFO: `/v3/users/6c625c8508504fe7ba38ed46ddab504f` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:19.248530 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c3e82c5-733f-436f-9dd4-46f282a4117f None admin] RBAC: Authorizing `identity:get_user(user_id=6c625c8508504fe7ba38ed46ddab504f)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:19.250943 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c3e82c5-733f-436f-9dd4-46f282a4117f None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:19.253808 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 129/249] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Aug 28 00:41:18 2022] GET /identity/v3/users/6c625c8508504fe7ba38ed46ddab504f => generated 297 bytes in 257 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:19.260594 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c3e82c5-733f-436f-9dd4-46f282a4117f None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:19.263372 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-393442e3-b37f-4c1b-8de7-bb459ae99b2b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:19.264114 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-393442e3-b37f-4c1b-8de7-bb459ae99b2b None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:19.264357 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-393442e3-b37f-4c1b-8de7-bb459ae99b2b None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:19.264623 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-393442e3-b37f-4c1b-8de7-bb459ae99b2b None admin] PATH_INFO: `/v3/users/cbabe7b652f54aa096188a7ed46cfeb0` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:19.266599 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-393442e3-b37f-4c1b-8de7-bb459ae99b2b None admin] RBAC: Authorizing `identity:get_user(user_id=cbabe7b652f54aa096188a7ed46cfeb0)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:19.267419 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-393442e3-b37f-4c1b-8de7-bb459ae99b2b None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:19.270026 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 121/250] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Aug 28 00:41:19 2022] GET /identity/v3/users/cbabe7b652f54aa096188a7ed46cfeb0 => generated 297 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:19.278113 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-393442e3-b37f-4c1b-8de7-bb459ae99b2b None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:19.322334 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:19.399969 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-393442e3-b37f-4c1b-8de7-bb459ae99b2b None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:19.401222 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:19.408153 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c3e82c5-733f-436f-9dd4-46f282a4117f None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:19.459862 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fdeb78c-5fd5-49ae-87ed-9d259ff8a92e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:19.461100 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fdeb78c-5fd5-49ae-87ed-9d259ff8a92e None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:19.461788 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fdeb78c-5fd5-49ae-87ed-9d259ff8a92e None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:19.462256 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fdeb78c-5fd5-49ae-87ed-9d259ff8a92e None admin] PATH_INFO: `/v3/projects/2635535fcc444c74b40bb9c46956c259` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:19.464009 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fdeb78c-5fd5-49ae-87ed-9d259ff8a92e None admin] RBAC: Authorizing `identity:get_project(project_id=2635535fcc444c74b40bb9c46956c259)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:19.466134 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fdeb78c-5fd5-49ae-87ed-9d259ff8a92e None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:19.468146 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 122/251] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Aug 28 00:41:19 2022] GET /identity/v3/projects/2635535fcc444c74b40bb9c46956c259 => generated 302 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:19.468990 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fa5ceaa-da12-4f6b-81d5-7e3c7f2718a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:19.469243 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fdeb78c-5fd5-49ae-87ed-9d259ff8a92e None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:19.470058 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fa5ceaa-da12-4f6b-81d5-7e3c7f2718a0 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:19.470536 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fa5ceaa-da12-4f6b-81d5-7e3c7f2718a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:19.471075 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fa5ceaa-da12-4f6b-81d5-7e3c7f2718a0 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:19.598132 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fa5ceaa-da12-4f6b-81d5-7e3c7f2718a0 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:19.599652 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fa5ceaa-da12-4f6b-81d5-7e3c7f2718a0 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:19.624799 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:19.629819 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fdeb78c-5fd5-49ae-87ed-9d259ff8a92e None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:19.667621 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3462bbc-ed70-45b2-b457-5c239ac81d90 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:19.668607 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3462bbc-ed70-45b2-b457-5c239ac81d90 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:19.668889 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3462bbc-ed70-45b2-b457-5c239ac81d90 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:19.669202 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3462bbc-ed70-45b2-b457-5c239ac81d90 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:19.680321 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3462bbc-ed70-45b2-b457-5c239ac81d90 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 28 00:41:19.685369 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-c3462bbc-ed70-45b2-b457-5c239ac81d90 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 28 00:41:19.685369 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:41:19.685369 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:41:19.685369 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:41:19.685369 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:41:19.685369 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:41:19.685369 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:41:19.685369 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:41:19.685369 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:41:19.685369 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:41:19.685369 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:41:19.685369 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:41:19.685369 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 28 00:41:19.685369 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 28 00:41:19.685369 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 28 00:41:19.685369 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 28 00:41:19.686300 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 28 00:41:19.686300 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 28 00:41:19.686300 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 28 00:41:19.686300 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 28 00:41:19.686300 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:41:19.686300 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:41:19.686300 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:41:19.686300 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:41:19.686300 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:41:19.686300 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:41:19.686300 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:41:19.686300 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:41:19.686300 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:41:19.686300 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:41:19.686300 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:41:19.686300 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:41:19.687089 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:41:19.687089 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:41:19.687089 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:41:19.687089 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:41:19.687089 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 28 00:41:19.687089 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 28 00:41:19.687089 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 28 00:41:19.687089 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 28 00:41:19.687089 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 28 00:41:19.687089 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 28 00:41:19.687089 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 28 00:41:19.687089 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:41:19.687089 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 123/252] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Aug 28 00:41:19 2022] GET /identity/v3/roles/admin => generated 83 bytes in 218 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:41:19.687824 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3462bbc-ed70-45b2-b457-5c239ac81d90 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:19.722129 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:19.731036 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 130/253] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Aug 28 00:41:19 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 466 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 28 00:41:19.732487 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d57a642-3f62-4467-8916-9929148712a3 None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:19.732780 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d57a642-3f62-4467-8916-9929148712a3 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:19.733088 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d57a642-3f62-4467-8916-9929148712a3 None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:19.739643 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 131/254] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:41:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:41:19.739643 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d57a642-3f62-4467-8916-9929148712a3 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:19.814041 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3462bbc-ed70-45b2-b457-5c239ac81d90 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:19.849863 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d57a642-3f62-4467-8916-9929148712a3 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:19.852855 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02ecee88-e92e-46f4-b916-113a13aad280 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:19.853758 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02ecee88-e92e-46f4-b916-113a13aad280 None admin] REQUEST_METHOD: `PUT` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:19.854213 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02ecee88-e92e-46f4-b916-113a13aad280 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:19.854682 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02ecee88-e92e-46f4-b916-113a13aad280 None admin] PATH_INFO: `/v3/projects/2635535fcc444c74b40bb9c46956c259/users/cbabe7b652f54aa096188a7ed46cfeb0/roles/0487ef7e943e4701915cb8ce0f2899d7` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:19.856839 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02ecee88-e92e-46f4-b916-113a13aad280 None admin] RBAC: Authorizing `identity:create_grant(project_id=2635535fcc444c74b40bb9c46956c259, user_id=cbabe7b652f54aa096188a7ed46cfeb0, role_id=0487ef7e943e4701915cb8ce0f2899d7)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:19.858701 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02ecee88-e92e-46f4-b916-113a13aad280 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:19.876748 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 124/255] 10.222.0.42 () {58 vars in 1439 bytes} [Sun Aug 28 00:41:19 2022] PUT /identity/v3/projects/2635535fcc444c74b40bb9c46956c259/users/cbabe7b652f54aa096188a7ed46cfeb0/roles/0487ef7e943e4701915cb8ce0f2899d7 => generated 0 bytes in 190 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:41:19.877992 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02ecee88-e92e-46f4-b916-113a13aad280 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:19.903719 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-869dfffd-7820-43d1-970f-d09a19d647a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:19.904722 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-869dfffd-7820-43d1-970f-d09a19d647a4 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:19.904722 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-869dfffd-7820-43d1-970f-d09a19d647a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:19.905002 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-869dfffd-7820-43d1-970f-d09a19d647a4 None admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:19.905701 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-869dfffd-7820-43d1-970f-d09a19d647a4 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:19.906003 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-869dfffd-7820-43d1-970f-d09a19d647a4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:19.907447 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-869dfffd-7820-43d1-970f-d09a19d647a4 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:19.909096 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:19.942127 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 132/256] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Aug 28 00:41:19 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:19.943822 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b692576-a8e7-4bba-a8ff-52994bd4b32f None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:19.944293 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b692576-a8e7-4bba-a8ff-52994bd4b32f None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:19.944744 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b692576-a8e7-4bba-a8ff-52994bd4b32f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:19.989358 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-6b692576-a8e7-4bba-a8ff-52994bd4b32f None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:41:19.990992 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-6b692576-a8e7-4bba-a8ff-52994bd4b32f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:41:20.080211 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02ecee88-e92e-46f4-b916-113a13aad280 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:20.091909 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:20.105775 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 133/257] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:41:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 28 00:41:20.107017 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b692576-a8e7-4bba-a8ff-52994bd4b32f None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:20.134661 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89cbfe2f-4cc5-4594-8b61-100b1597dca9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:20.135299 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89cbfe2f-4cc5-4594-8b61-100b1597dca9 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:20.135576 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89cbfe2f-4cc5-4594-8b61-100b1597dca9 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:20.135846 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89cbfe2f-4cc5-4594-8b61-100b1597dca9 None admin] PATH_INFO: `/v3/projects/2635535fcc444c74b40bb9c46956c259` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:20.138083 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89cbfe2f-4cc5-4594-8b61-100b1597dca9 None admin] RBAC: Authorizing `identity:get_project(project_id=2635535fcc444c74b40bb9c46956c259)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:20.140185 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89cbfe2f-4cc5-4594-8b61-100b1597dca9 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:20.142485 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 125/258] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Aug 28 00:41:19 2022] GET /identity/v3/projects/2635535fcc444c74b40bb9c46956c259 => generated 302 bytes in 265 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:20.149278 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89cbfe2f-4cc5-4594-8b61-100b1597dca9 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:20.252064 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b692576-a8e7-4bba-a8ff-52994bd4b32f None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:20.308303 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b27a91a-e4e2-4868-b40a-64dc57e0ff8d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:20.309169 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b27a91a-e4e2-4868-b40a-64dc57e0ff8d None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:20.309457 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b27a91a-e4e2-4868-b40a-64dc57e0ff8d None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:20.309735 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b27a91a-e4e2-4868-b40a-64dc57e0ff8d None admin] PATH_INFO: `/v3/users/6e82746467384e4e9ec43cf94e3a5040` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:20.314082 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b27a91a-e4e2-4868-b40a-64dc57e0ff8d None admin] RBAC: Authorizing `identity:get_user(user_id=6e82746467384e4e9ec43cf94e3a5040)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:20.324151 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b27a91a-e4e2-4868-b40a-64dc57e0ff8d None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:20.332429 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 134/259] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Aug 28 00:41:20 2022] GET /identity/v3/users/6e82746467384e4e9ec43cf94e3a5040 => generated 283 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:20.333094 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b27a91a-e4e2-4868-b40a-64dc57e0ff8d None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:20.386246 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:20.417440 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89cbfe2f-4cc5-4594-8b61-100b1597dca9 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:20.472646 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:20.501917 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4c92fe9-d10e-46b6-9a4f-0da3b35d5810 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:20.502601 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4c92fe9-d10e-46b6-9a4f-0da3b35d5810 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:20.502820 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4c92fe9-d10e-46b6-9a4f-0da3b35d5810 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:20.503091 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4c92fe9-d10e-46b6-9a4f-0da3b35d5810 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:20.503831 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4c92fe9-d10e-46b6-9a4f-0da3b35d5810 None admin] RBAC: Adding query filter params (role.id=60823fd2c27d46e98b5d299e86b51918, scope.project.id=2635535fcc444c74b40bb9c46956c259, user.id=6c625c8508504fe7ba38ed46ddab504f) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:20.504142 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4c92fe9-d10e-46b6-9a4f-0da3b35d5810 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:20.505718 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4c92fe9-d10e-46b6-9a4f-0da3b35d5810 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:20.518429 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 126/260] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Aug 28 00:41:20 2022] GET /identity/v3/role_assignments?user.id=6c625c8508504fe7ba38ed46ddab504f&scope.project.id=2635535fcc444c74b40bb9c46956c259&role.id=60823fd2c27d46e98b5d299e86b51918 => generated 259 bytes in 365 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:20.519706 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4c92fe9-d10e-46b6-9a4f-0da3b35d5810 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:20.556615 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b27a91a-e4e2-4868-b40a-64dc57e0ff8d None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:20.633282 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e15fd692-36bd-4167-83ad-831e2b2dc48b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:20.633916 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e15fd692-36bd-4167-83ad-831e2b2dc48b None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:20.634196 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e15fd692-36bd-4167-83ad-831e2b2dc48b None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:20.634469 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e15fd692-36bd-4167-83ad-831e2b2dc48b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:20.733639 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4c92fe9-d10e-46b6-9a4f-0da3b35d5810 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:20.785341 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9536fb9-3c1e-4681-bf67-fc4e2547de26 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:20.786319 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9536fb9-3c1e-4681-bf67-fc4e2547de26 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:20.786799 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9536fb9-3c1e-4681-bf67-fc4e2547de26 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:20.787283 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9536fb9-3c1e-4681-bf67-fc4e2547de26 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:20.804309 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9536fb9-3c1e-4681-bf67-fc4e2547de26 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 28 00:41:20.810215 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-a9536fb9-3c1e-4681-bf67-fc4e2547de26 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 28 00:41:20.810215 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:41:20.810215 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:41:20.810215 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:41:20.810215 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:41:20.810215 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:41:20.810215 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:41:20.810215 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:41:20.810215 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:41:20.810215 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:41:20.810215 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:41:20.810215 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:41:20.810215 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 28 00:41:20.810215 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 28 00:41:20.810215 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 28 00:41:20.810215 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 28 00:41:20.811127 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 28 00:41:20.811127 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 28 00:41:20.811127 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 28 00:41:20.811127 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 28 00:41:20.811127 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:41:20.811127 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:41:20.811127 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:41:20.811127 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:41:20.811127 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:41:20.811127 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:41:20.811127 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:41:20.811127 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:41:20.811127 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:41:20.811127 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:41:20.811127 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:41:20.811127 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:41:20.811935 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:41:20.811935 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:41:20.811935 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:41:20.811935 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:41:20.811935 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 28 00:41:20.811935 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 28 00:41:20.811935 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 28 00:41:20.811935 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 28 00:41:20.811935 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 28 00:41:20.811935 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 28 00:41:20.811935 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 28 00:41:20.811935 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:41:20.818856 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 127/261] 10.222.0.42 () {56 vars in 1114 bytes} [Sun Aug 28 00:41:20 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 300 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:41:20.824854 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9536fb9-3c1e-4681-bf67-fc4e2547de26 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:20.845419 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:20.891432 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e15fd692-36bd-4167-83ad-831e2b2dc48b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:20.892946 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e15fd692-36bd-4167-83ad-831e2b2dc48b None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:20.967813 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9536fb9-3c1e-4681-bf67-fc4e2547de26 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:21.031178 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d39a211e-b984-48b3-8173-e273139a2161 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:21.032250 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d39a211e-b984-48b3-8173-e273139a2161 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:21.032730 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d39a211e-b984-48b3-8173-e273139a2161 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:21.033282 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d39a211e-b984-48b3-8173-e273139a2161 None admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:21.034158 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d39a211e-b984-48b3-8173-e273139a2161 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:21.034761 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d39a211e-b984-48b3-8173-e273139a2161 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:21.036424 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d39a211e-b984-48b3-8173-e273139a2161 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:21.041337 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:21.063211 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 128/262] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Aug 28 00:41:20 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 239 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:21.068086 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d39a211e-b984-48b3-8173-e273139a2161 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:21.152680 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 135/263] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Aug 28 00:41:20 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 820 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 28 00:41:21.158778 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e15fd692-36bd-4167-83ad-831e2b2dc48b None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:21.194207 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:21.238176 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d39a211e-b984-48b3-8173-e273139a2161 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:21.286137 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1726e12f-df26-4708-b03c-6a267eaac6c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:21.286842 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1726e12f-df26-4708-b03c-6a267eaac6c7 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:21.287065 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1726e12f-df26-4708-b03c-6a267eaac6c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:21.287341 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1726e12f-df26-4708-b03c-6a267eaac6c7 None admin] PATH_INFO: `/v3/users/6e82746467384e4e9ec43cf94e3a5040` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:21.288486 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1726e12f-df26-4708-b03c-6a267eaac6c7 None admin] RBAC: Authorizing `identity:get_user(user_id=6e82746467384e4e9ec43cf94e3a5040)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:21.290848 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1726e12f-df26-4708-b03c-6a267eaac6c7 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:21.293882 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 129/264] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Aug 28 00:41:21 2022] GET /identity/v3/users/6e82746467384e4e9ec43cf94e3a5040 => generated 283 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:21.295244 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66888844-dbea-4d6d-8f30-5b11c46e2c41 None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:21.295984 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66888844-dbea-4d6d-8f30-5b11c46e2c41 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:21.296269 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66888844-dbea-4d6d-8f30-5b11c46e2c41 None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:21.297896 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 130/265] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:41:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:41:21.305930 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cefc7e33-0d59-4b2e-a301-4cac37e3fc14 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:21.306442 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cefc7e33-0d59-4b2e-a301-4cac37e3fc14 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:21.306874 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cefc7e33-0d59-4b2e-a301-4cac37e3fc14 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:21.340707 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-cefc7e33-0d59-4b2e-a301-4cac37e3fc14 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:41:21.341957 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-cefc7e33-0d59-4b2e-a301-4cac37e3fc14 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:41:21.419937 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e15fd692-36bd-4167-83ad-831e2b2dc48b None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:21.461677 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eec0666-e686-471f-b366-c71b9185b09f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:21.462332 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eec0666-e686-471f-b366-c71b9185b09f None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:21.462608 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eec0666-e686-471f-b366-c71b9185b09f None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:21.462881 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eec0666-e686-471f-b366-c71b9185b09f None admin] PATH_INFO: `/v3/projects/2635535fcc444c74b40bb9c46956c259` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:21.465190 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3eec0666-e686-471f-b366-c71b9185b09f None admin] RBAC: Authorizing `identity:get_project(project_id=2635535fcc444c74b40bb9c46956c259)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:21.466654 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3eec0666-e686-471f-b366-c71b9185b09f None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:21.468307 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 136/266] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Aug 28 00:41:21 2022] GET /identity/v3/projects/2635535fcc444c74b40bb9c46956c259 => generated 302 bytes in 310 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:21.474309 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eec0666-e686-471f-b366-c71b9185b09f None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:21.494245 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 131/267] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:41:21 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 28 00:41:21.494958 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cefc7e33-0d59-4b2e-a301-4cac37e3fc14 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:21.535213 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:21.648354 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:21.685741 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eec0666-e686-471f-b366-c71b9185b09f None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:21.719711 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cefc7e33-0d59-4b2e-a301-4cac37e3fc14 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:21.745285 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-182b92f5-ebca-4ca5-99eb-f24e4166bb6c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:21.745904 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-182b92f5-ebca-4ca5-99eb-f24e4166bb6c None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:21.746183 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-182b92f5-ebca-4ca5-99eb-f24e4166bb6c None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:21.746456 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-182b92f5-ebca-4ca5-99eb-f24e4166bb6c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:21.791150 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f63bc471-e432-4ec9-87fb-53bd31757fcb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:21.791772 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f63bc471-e432-4ec9-87fb-53bd31757fcb None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:21.792047 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f63bc471-e432-4ec9-87fb-53bd31757fcb None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:21.792316 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f63bc471-e432-4ec9-87fb-53bd31757fcb None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:21.793069 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f63bc471-e432-4ec9-87fb-53bd31757fcb None admin] RBAC: Adding query filter params (role.id=f08d6043261747e3b4763c744eef7dfb, scope.project.id=2635535fcc444c74b40bb9c46956c259, user.id=6e82746467384e4e9ec43cf94e3a5040) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:21.793375 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f63bc471-e432-4ec9-87fb-53bd31757fcb None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:21.796028 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f63bc471-e432-4ec9-87fb-53bd31757fcb None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:21.808458 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 132/268] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Aug 28 00:41:21 2022] GET /identity/v3/role_assignments?user.id=6e82746467384e4e9ec43cf94e3a5040&scope.project.id=2635535fcc444c74b40bb9c46956c259&role.id=f08d6043261747e3b4763c744eef7dfb => generated 608 bytes in 314 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:21.810569 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3dd210c-3ad4-4629-b343-2172544d5c22 None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:21.810849 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3dd210c-3ad4-4629-b343-2172544d5c22 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:21.811121 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3dd210c-3ad4-4629-b343-2172544d5c22 None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:21.812547 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 133/269] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:41:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:41:21.826527 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b43ab164-bdc1-4bec-8d1f-b946f73028c2 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:21.826841 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b43ab164-bdc1-4bec-8d1f-b946f73028c2 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:21.827070 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b43ab164-bdc1-4bec-8d1f-b946f73028c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:21.868281 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-b43ab164-bdc1-4bec-8d1f-b946f73028c2 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:41:21.869534 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-b43ab164-bdc1-4bec-8d1f-b946f73028c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:41:21.899251 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-182b92f5-ebca-4ca5-99eb-f24e4166bb6c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:21.900666 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-182b92f5-ebca-4ca5-99eb-f24e4166bb6c None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:21.960058 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:21.999162 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 134/270] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:41:21 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 28 00:41:22.000127 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b43ab164-bdc1-4bec-8d1f-b946f73028c2 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:22.095311 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:22.111065 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 137/271] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Aug 28 00:41:21 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 642 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 28 00:41:22.124826 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-182b92f5-ebca-4ca5-99eb-f24e4166bb6c None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:22.175173 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b43ab164-bdc1-4bec-8d1f-b946f73028c2 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:22.237017 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f571ec5c-e55d-4bf5-b585-92487da8a42c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:22.237669 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f571ec5c-e55d-4bf5-b585-92487da8a42c None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:22.237945 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f571ec5c-e55d-4bf5-b585-92487da8a42c None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:22.238216 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f571ec5c-e55d-4bf5-b585-92487da8a42c None admin] PATH_INFO: `/v3/roles/member` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:22.262741 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f571ec5c-e55d-4bf5-b585-92487da8a42c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 28 00:41:22.267967 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-f571ec5c-e55d-4bf5-b585-92487da8a42c None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 28 00:41:22.267967 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:41:22.267967 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:41:22.267967 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:41:22.267967 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:41:22.267967 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:41:22.267967 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:41:22.267967 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:41:22.267967 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:41:22.267967 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:41:22.267967 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:41:22.267967 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:41:22.267967 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 28 00:41:22.267967 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 28 00:41:22.267967 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 28 00:41:22.267967 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 28 00:41:22.268842 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 28 00:41:22.268842 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 28 00:41:22.268842 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 28 00:41:22.268842 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 28 00:41:22.268842 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:41:22.268842 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:41:22.268842 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:41:22.268842 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:41:22.268842 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:41:22.268842 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:41:22.268842 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:41:22.268842 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:41:22.268842 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:41:22.268842 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:41:22.268842 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:41:22.268842 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:41:22.269825 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:41:22.269825 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:41:22.269825 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:41:22.269825 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:41:22.269825 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 28 00:41:22.269825 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 28 00:41:22.269825 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 28 00:41:22.269825 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 28 00:41:22.269825 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 28 00:41:22.269825 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 28 00:41:22.269825 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 28 00:41:22.269825 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:41:22.269825 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 135/272] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Aug 28 00:41:21 2022] GET /identity/v3/roles/member => generated 84 bytes in 270 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:41:22.279255 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f571ec5c-e55d-4bf5-b585-92487da8a42c None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:22.314307 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-182b92f5-ebca-4ca5-99eb-f24e4166bb6c None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:22.359665 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:22.372206 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddd600d7-43e1-4521-8647-5c517cc45d94 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:22.373000 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddd600d7-43e1-4521-8647-5c517cc45d94 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:22.373272 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddd600d7-43e1-4521-8647-5c517cc45d94 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:22.373550 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddd600d7-43e1-4521-8647-5c517cc45d94 None admin] PATH_INFO: `/v3/projects/2635535fcc444c74b40bb9c46956c259` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:22.374713 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddd600d7-43e1-4521-8647-5c517cc45d94 None admin] RBAC: Authorizing `identity:get_project(project_id=2635535fcc444c74b40bb9c46956c259)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:22.376354 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddd600d7-43e1-4521-8647-5c517cc45d94 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:22.378576 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 138/273] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Aug 28 00:41:22 2022] GET /identity/v3/projects/2635535fcc444c74b40bb9c46956c259 => generated 302 bytes in 254 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:22.379633 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddd600d7-43e1-4521-8647-5c517cc45d94 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:22.438660 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f571ec5c-e55d-4bf5-b585-92487da8a42c None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:22.465702 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:22.479751 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7bc1aa2-d584-4025-b7cf-d1560f1f1178 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:22.480999 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7bc1aa2-d584-4025-b7cf-d1560f1f1178 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:22.481549 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7bc1aa2-d584-4025-b7cf-d1560f1f1178 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:22.481991 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7bc1aa2-d584-4025-b7cf-d1560f1f1178 None admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:22.482908 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7bc1aa2-d584-4025-b7cf-d1560f1f1178 None admin] RBAC: Adding query filter params (name=member) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:22.483354 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7bc1aa2-d584-4025-b7cf-d1560f1f1178 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:22.485158 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7bc1aa2-d584-4025-b7cf-d1560f1f1178 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:22.495278 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 136/274] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Aug 28 00:41:22 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 28 00:41:22.500803 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7bc1aa2-d584-4025-b7cf-d1560f1f1178 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:22.508105 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddd600d7-43e1-4521-8647-5c517cc45d94 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:22.543899 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab0f7320-d21d-43be-8f4d-b513cddf6f5c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:22.544812 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab0f7320-d21d-43be-8f4d-b513cddf6f5c None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:22.545256 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab0f7320-d21d-43be-8f4d-b513cddf6f5c None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:22.545682 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab0f7320-d21d-43be-8f4d-b513cddf6f5c None admin] PATH_INFO: `/v3/domains/default` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:22.546806 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab0f7320-d21d-43be-8f4d-b513cddf6f5c None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:22.548290 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab0f7320-d21d-43be-8f4d-b513cddf6f5c None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:22.550123 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 139/275] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:41:22 2022] GET /identity/v3/domains/default => generated 199 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:22.551062 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab0f7320-d21d-43be-8f4d-b513cddf6f5c None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:22.617353 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:22.635023 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7bc1aa2-d584-4025-b7cf-d1560f1f1178 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:22.660825 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab0f7320-d21d-43be-8f4d-b513cddf6f5c None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:22.672611 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18ab54f8-e7da-4b4a-bee3-4f0e6c061a93 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:22.673516 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18ab54f8-e7da-4b4a-bee3-4f0e6c061a93 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:22.673955 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18ab54f8-e7da-4b4a-bee3-4f0e6c061a93 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:22.674419 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18ab54f8-e7da-4b4a-bee3-4f0e6c061a93 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:22.675379 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18ab54f8-e7da-4b4a-bee3-4f0e6c061a93 None admin] RBAC: Adding query filter params (role.id=05caf0846a15452b9a6a79fa59c7fa5a, scope.project.id=2635535fcc444c74b40bb9c46956c259, user.id=6e82746467384e4e9ec43cf94e3a5040) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:22.675827 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18ab54f8-e7da-4b4a-bee3-4f0e6c061a93 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:22.677674 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18ab54f8-e7da-4b4a-bee3-4f0e6c061a93 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:22.684195 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 137/276] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Aug 28 00:41:22 2022] GET /identity/v3/role_assignments?user.id=6e82746467384e4e9ec43cf94e3a5040&scope.project.id=2635535fcc444c74b40bb9c46956c259&role.id=05caf0846a15452b9a6a79fa59c7fa5a => generated 608 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:22.685111 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18ab54f8-e7da-4b4a-bee3-4f0e6c061a93 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:22.703492 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18c66bdf-80d4-4cb0-8816-1b29f7b07961 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:22.705289 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18c66bdf-80d4-4cb0-8816-1b29f7b07961 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:22.706025 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18c66bdf-80d4-4cb0-8816-1b29f7b07961 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:22.706483 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18c66bdf-80d4-4cb0-8816-1b29f7b07961 None admin] PATH_INFO: `/v3/users/267d82a511ef46d884fe5913687155ca` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:22.726074 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18c66bdf-80d4-4cb0-8816-1b29f7b07961 None admin] RBAC: Authorizing `identity:get_user(user_id=267d82a511ef46d884fe5913687155ca)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:22.728937 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18c66bdf-80d4-4cb0-8816-1b29f7b07961 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:22.731008 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 140/277] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Aug 28 00:41:22 2022] GET /identity/v3/users/267d82a511ef46d884fe5913687155ca => generated 293 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:22.732595 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0430310-6d2a-4ac0-aba9-611384f6c742 None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:22.733130 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0430310-6d2a-4ac0-aba9-611384f6c742 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:22.733601 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0430310-6d2a-4ac0-aba9-611384f6c742 None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:22.740343 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 141/278] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:41:22 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:41:22.741691 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0430310-6d2a-4ac0-aba9-611384f6c742 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:22.752538 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:22.812195 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18ab54f8-e7da-4b4a-bee3-4f0e6c061a93 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:22.857157 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4184eb21-a042-4008-8454-b2fc5aab6559 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:22.857931 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4184eb21-a042-4008-8454-b2fc5aab6559 None admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:22.858873 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4184eb21-a042-4008-8454-b2fc5aab6559 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:22.858873 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4184eb21-a042-4008-8454-b2fc5aab6559 None admin] PATH_INFO: `/v3/users` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:22.859808 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4184eb21-a042-4008-8454-b2fc5aab6559 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:22.862127 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4184eb21-a042-4008-8454-b2fc5aab6559 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:22.873067 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0430310-6d2a-4ac0-aba9-611384f6c742 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:22.889819 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 138/279] 10.222.0.42 () {60 vars in 1132 bytes} [Sun Aug 28 00:41:22 2022] POST /identity/v3/users => generated 293 bytes in 205 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:41:22.891240 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e12d14a4-923e-40d2-b75f-ff3f01935876 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:22.891537 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e12d14a4-923e-40d2-b75f-ff3f01935876 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:22.891815 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e12d14a4-923e-40d2-b75f-ff3f01935876 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:22.905988 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:22.914718 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-e12d14a4-923e-40d2-b75f-ff3f01935876 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:41:22.914987 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-e12d14a4-923e-40d2-b75f-ff3f01935876 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:41:22.922632 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b105131a-5370-4a27-818d-158439301759 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:22.923329 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b105131a-5370-4a27-818d-158439301759 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:22.923615 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b105131a-5370-4a27-818d-158439301759 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:22.924519 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b105131a-5370-4a27-818d-158439301759 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:22.925447 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b105131a-5370-4a27-818d-158439301759 None admin] RBAC: Adding query filter params (role.id=0487ef7e943e4701915cb8ce0f2899d7, user.id=267d82a511ef46d884fe5913687155ca, scope.system=all) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:22.925759 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b105131a-5370-4a27-818d-158439301759 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:22.927312 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b105131a-5370-4a27-818d-158439301759 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:22.933704 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 142/280] 10.222.0.42 () {56 vars in 1308 bytes} [Sun Aug 28 00:41:22 2022] GET /identity/v3/role_assignments?user.id=267d82a511ef46d884fe5913687155ca&scope.system=all&role.id=0487ef7e943e4701915cb8ce0f2899d7 => generated 226 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:22.935106 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f1715b6-c845-4879-b6b7-2af9d8c05439 None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:22.935400 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f1715b6-c845-4879-b6b7-2af9d8c05439 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:22.935681 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f1715b6-c845-4879-b6b7-2af9d8c05439 None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:22.937260 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 143/281] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:41:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:41:22.943467 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a648ef35-3407-4111-a50a-267f675805fb None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:22.944358 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a648ef35-3407-4111-a50a-267f675805fb None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:22.944358 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a648ef35-3407-4111-a50a-267f675805fb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:22.967777 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-a648ef35-3407-4111-a50a-267f675805fb None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:41:22.970333 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-a648ef35-3407-4111-a50a-267f675805fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:41:23.033650 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 139/282] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:41:22 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 28 00:41:23.065923 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 144/283] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:41:22 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 28 00:41:23.313431 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e12d14a4-923e-40d2-b75f-ff3f01935876 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:23.349948 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:23.368644 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a648ef35-3407-4111-a50a-267f675805fb None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:23.450876 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e12d14a4-923e-40d2-b75f-ff3f01935876 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:23.488186 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-996bc42b-18e4-4a2a-b395-685f7ebe3352 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:23.488899 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-996bc42b-18e4-4a2a-b395-685f7ebe3352 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:23.489253 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-996bc42b-18e4-4a2a-b395-685f7ebe3352 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:23.489527 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-996bc42b-18e4-4a2a-b395-685f7ebe3352 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:23.504991 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-996bc42b-18e4-4a2a-b395-685f7ebe3352 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 28 00:41:23.505594 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-996bc42b-18e4-4a2a-b395-685f7ebe3352 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 28 00:41:23.505594 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:41:23.505594 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:41:23.505594 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:41:23.505594 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:41:23.505594 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:41:23.505594 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:41:23.505594 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:41:23.505594 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:41:23.505594 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:41:23.505594 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:41:23.505594 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:41:23.505594 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 28 00:41:23.505594 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 28 00:41:23.505594 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 28 00:41:23.505594 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 28 00:41:23.506437 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 28 00:41:23.506437 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 28 00:41:23.506437 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 28 00:41:23.506437 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 28 00:41:23.506437 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:41:23.506437 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:41:23.506437 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:41:23.506437 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:41:23.506437 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:41:23.506437 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:41:23.506437 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:41:23.506437 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:41:23.506437 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:41:23.506437 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:41:23.506437 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:41:23.506437 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:41:23.507404 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:41:23.507404 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:41:23.507404 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:41:23.507404 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:41:23.507404 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 28 00:41:23.507404 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 28 00:41:23.507404 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 28 00:41:23.507404 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 28 00:41:23.507404 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 28 00:41:23.507404 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 28 00:41:23.507404 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 28 00:41:23.507404 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:41:23.507404 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 140/284] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Aug 28 00:41:23 2022] GET /identity/v3/roles/member => generated 84 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:41:23.513060 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-996bc42b-18e4-4a2a-b395-685f7ebe3352 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:23.566009 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a648ef35-3407-4111-a50a-267f675805fb None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:23.611855 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:23.636615 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fe79e39-bdad-4078-b31c-152a2f7a64a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:23.637295 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fe79e39-bdad-4078-b31c-152a2f7a64a8 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:23.637575 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fe79e39-bdad-4078-b31c-152a2f7a64a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:23.637845 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fe79e39-bdad-4078-b31c-152a2f7a64a8 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:23.649181 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fe79e39-bdad-4078-b31c-152a2f7a64a8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 28 00:41:23.649992 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:23.656185 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-6fe79e39-bdad-4078-b31c-152a2f7a64a8 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 28 00:41:23.656185 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:41:23.656185 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:41:23.656185 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:41:23.656185 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:41:23.656185 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:41:23.656185 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:41:23.656185 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:41:23.656185 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:41:23.656185 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:41:23.656185 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:41:23.656185 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:41:23.656185 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 28 00:41:23.656185 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 28 00:41:23.656185 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 28 00:41:23.656185 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 28 00:41:23.657132 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 28 00:41:23.657132 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 28 00:41:23.657132 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 28 00:41:23.657132 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 28 00:41:23.657132 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:41:23.657132 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:41:23.657132 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:41:23.657132 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:41:23.657132 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:41:23.657132 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:41:23.657132 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:41:23.657132 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:41:23.657132 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:41:23.657132 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:41:23.657132 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:41:23.657132 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:41:23.657932 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:41:23.657932 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:41:23.657932 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:41:23.657932 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:41:23.657932 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 28 00:41:23.657932 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 28 00:41:23.657932 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 28 00:41:23.657932 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 28 00:41:23.657932 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 28 00:41:23.657932 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 28 00:41:23.657932 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 28 00:41:23.657932 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:41:23.657932 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 145/285] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Aug 28 00:41:23 2022] GET /identity/v3/roles/reader => generated 84 bytes in 289 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 28 00:41:23.663613 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fe79e39-bdad-4078-b31c-152a2f7a64a8 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:23.667854 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-996bc42b-18e4-4a2a-b395-685f7ebe3352 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:23.763318 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56924609-e703-49ee-9b48-a3924f572967 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:23.763930 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56924609-e703-49ee-9b48-a3924f572967 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:23.764209 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56924609-e703-49ee-9b48-a3924f572967 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:23.764472 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56924609-e703-49ee-9b48-a3924f572967 None admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:23.765169 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56924609-e703-49ee-9b48-a3924f572967 None admin] RBAC: Adding query filter params (name=member) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:23.765468 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56924609-e703-49ee-9b48-a3924f572967 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:23.766925 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56924609-e703-49ee-9b48-a3924f572967 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:23.777535 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 141/286] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Aug 28 00:41:23 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 265 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:23.785966 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56924609-e703-49ee-9b48-a3924f572967 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:23.791749 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fe79e39-bdad-4078-b31c-152a2f7a64a8 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:23.892684 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56924609-e703-49ee-9b48-a3924f572967 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:23.910227 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84b6df6b-0651-4418-915b-364ca6a38dc3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:23.910850 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84b6df6b-0651-4418-915b-364ca6a38dc3 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:23.911134 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84b6df6b-0651-4418-915b-364ca6a38dc3 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:23.911407 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84b6df6b-0651-4418-915b-364ca6a38dc3 None admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:23.912089 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84b6df6b-0651-4418-915b-364ca6a38dc3 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:23.912396 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84b6df6b-0651-4418-915b-364ca6a38dc3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:23.913823 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84b6df6b-0651-4418-915b-364ca6a38dc3 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:23.931036 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 146/287] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Aug 28 00:41:23 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 265 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 28 00:41:23.936605 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84b6df6b-0651-4418-915b-364ca6a38dc3 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:23.948521 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2ef0cc9-94eb-4da8-9c56-44e0b12c004d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:23.949205 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2ef0cc9-94eb-4da8-9c56-44e0b12c004d None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:23.949486 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2ef0cc9-94eb-4da8-9c56-44e0b12c004d None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:23.949762 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2ef0cc9-94eb-4da8-9c56-44e0b12c004d None admin] PATH_INFO: `/v3/users/cbabe7b652f54aa096188a7ed46cfeb0` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:23.952742 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2ef0cc9-94eb-4da8-9c56-44e0b12c004d None admin] RBAC: Authorizing `identity:get_user(user_id=cbabe7b652f54aa096188a7ed46cfeb0)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:23.954258 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2ef0cc9-94eb-4da8-9c56-44e0b12c004d None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:23.956298 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 142/288] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Aug 28 00:41:23 2022] GET /identity/v3/users/cbabe7b652f54aa096188a7ed46cfeb0 => generated 297 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:23.962051 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2ef0cc9-94eb-4da8-9c56-44e0b12c004d None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:24.005201 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:24.018722 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:24.123462 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84b6df6b-0651-4418-915b-364ca6a38dc3 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:24.136063 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2ef0cc9-94eb-4da8-9c56-44e0b12c004d None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:24.194066 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fd791d0-43a6-47c6-b3f0-80cf7f81f3c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:24.194635 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fd791d0-43a6-47c6-b3f0-80cf7f81f3c9 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:24.194915 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fd791d0-43a6-47c6-b3f0-80cf7f81f3c9 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:24.195188 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fd791d0-43a6-47c6-b3f0-80cf7f81f3c9 None admin] PATH_INFO: `/v3/users/6c625c8508504fe7ba38ed46ddab504f` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:24.196771 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fd791d0-43a6-47c6-b3f0-80cf7f81f3c9 None admin] RBAC: Authorizing `identity:get_user(user_id=6c625c8508504fe7ba38ed46ddab504f)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:24.198254 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fd791d0-43a6-47c6-b3f0-80cf7f81f3c9 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:24.199852 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 147/289] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Aug 28 00:41:23 2022] GET /identity/v3/users/6c625c8508504fe7ba38ed46ddab504f => generated 297 bytes in 264 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:24.206468 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fd791d0-43a6-47c6-b3f0-80cf7f81f3c9 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:24.211116 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c404178-1886-4a3c-ac3a-db00eb2dd248 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:24.211741 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c404178-1886-4a3c-ac3a-db00eb2dd248 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:24.212020 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c404178-1886-4a3c-ac3a-db00eb2dd248 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:24.212294 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c404178-1886-4a3c-ac3a-db00eb2dd248 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:24.321170 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c404178-1886-4a3c-ac3a-db00eb2dd248 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:24.322707 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c404178-1886-4a3c-ac3a-db00eb2dd248 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:24.371444 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:24.437860 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fd791d0-43a6-47c6-b3f0-80cf7f81f3c9 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:24.449794 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:24.468132 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 143/290] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Aug 28 00:41:23 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 506 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Aug 28 00:41:24.474186 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c404178-1886-4a3c-ac3a-db00eb2dd248 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:24.504603 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d67e6193-759a-44cb-9f89-8eb7cc6f0e9c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:24.505263 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d67e6193-759a-44cb-9f89-8eb7cc6f0e9c None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:24.505540 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d67e6193-759a-44cb-9f89-8eb7cc6f0e9c None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:24.505809 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d67e6193-759a-44cb-9f89-8eb7cc6f0e9c None admin] PATH_INFO: `/v3/projects/2635535fcc444c74b40bb9c46956c259` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:24.507923 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d67e6193-759a-44cb-9f89-8eb7cc6f0e9c None admin] RBAC: Authorizing `identity:get_project(project_id=2635535fcc444c74b40bb9c46956c259)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:24.509404 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d67e6193-759a-44cb-9f89-8eb7cc6f0e9c None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:24.511055 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 148/291] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Aug 28 00:41:24 2022] GET /identity/v3/projects/2635535fcc444c74b40bb9c46956c259 => generated 302 bytes in 305 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:24.516642 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d67e6193-759a-44cb-9f89-8eb7cc6f0e9c None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:24.660593 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c404178-1886-4a3c-ac3a-db00eb2dd248 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:24.698498 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d67e6193-759a-44cb-9f89-8eb7cc6f0e9c None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:24.725631 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:24.732249 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0ab67c1-2e6d-4307-8096-281e303feaac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:24.734052 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ab67c1-2e6d-4307-8096-281e303feaac None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:24.734376 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ab67c1-2e6d-4307-8096-281e303feaac None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:24.734682 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ab67c1-2e6d-4307-8096-281e303feaac None admin] PATH_INFO: `/v3/projects/2635535fcc444c74b40bb9c46956c259` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:24.739088 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0ab67c1-2e6d-4307-8096-281e303feaac None admin] RBAC: Authorizing `identity:get_project(project_id=2635535fcc444c74b40bb9c46956c259)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:24.740634 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0ab67c1-2e6d-4307-8096-281e303feaac None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:24.742353 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 144/292] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Aug 28 00:41:24 2022] GET /identity/v3/projects/2635535fcc444c74b40bb9c46956c259 => generated 302 bytes in 269 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:24.748440 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0ab67c1-2e6d-4307-8096-281e303feaac None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:24.772066 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb600cb8-02d0-4faf-8f29-ecb7474329fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:24.772740 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb600cb8-02d0-4faf-8f29-ecb7474329fe None admin] REQUEST_METHOD: `PUT` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:24.773050 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb600cb8-02d0-4faf-8f29-ecb7474329fe None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:24.773322 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb600cb8-02d0-4faf-8f29-ecb7474329fe None admin] PATH_INFO: `/v3/projects/2635535fcc444c74b40bb9c46956c259/users/6c625c8508504fe7ba38ed46ddab504f/roles/60823fd2c27d46e98b5d299e86b51918` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:24.776779 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb600cb8-02d0-4faf-8f29-ecb7474329fe None admin] RBAC: Authorizing `identity:create_grant(project_id=2635535fcc444c74b40bb9c46956c259, user_id=6c625c8508504fe7ba38ed46ddab504f, role_id=60823fd2c27d46e98b5d299e86b51918)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:24.778268 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb600cb8-02d0-4faf-8f29-ecb7474329fe None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:24.794089 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 149/293] 10.222.0.42 () {58 vars in 1439 bytes} [Sun Aug 28 00:41:24 2022] PUT /identity/v3/projects/2635535fcc444c74b40bb9c46956c259/users/6c625c8508504fe7ba38ed46ddab504f/roles/60823fd2c27d46e98b5d299e86b51918 => generated 0 bytes in 278 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 28 00:41:24.914453 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0ab67c1-2e6d-4307-8096-281e303feaac None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:24.954056 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8380693-09c7-4fcf-87db-32c69cb05aff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:24.954692 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8380693-09c7-4fcf-87db-32c69cb05aff None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:24.955293 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8380693-09c7-4fcf-87db-32c69cb05aff None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:24.955566 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8380693-09c7-4fcf-87db-32c69cb05aff None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:24.956322 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8380693-09c7-4fcf-87db-32c69cb05aff None admin] RBAC: Adding query filter params (role.id=0487ef7e943e4701915cb8ce0f2899d7, scope.project.id=2635535fcc444c74b40bb9c46956c259, user.id=cbabe7b652f54aa096188a7ed46cfeb0) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:24.956632 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8380693-09c7-4fcf-87db-32c69cb05aff None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:24.958186 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8380693-09c7-4fcf-87db-32c69cb05aff None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:24.965334 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 145/294] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Aug 28 00:41:24 2022] GET /identity/v3/role_assignments?user.id=cbabe7b652f54aa096188a7ed46cfeb0&scope.project.id=2635535fcc444c74b40bb9c46956c259&role.id=0487ef7e943e4701915cb8ce0f2899d7 => generated 608 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:25.158074 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce41ab5-8b89-4caa-bb6f-1289db902d86 None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:25.158576 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce41ab5-8b89-4caa-bb6f-1289db902d86 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:25.158649 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce41ab5-8b89-4caa-bb6f-1289db902d86 None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:25.160099 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 150/295] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:41:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:41:25.165957 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5236005f-3b84-4f22-aff2-c80af311ab3e None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:25.166567 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5236005f-3b84-4f22-aff2-c80af311ab3e None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:25.167043 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5236005f-3b84-4f22-aff2-c80af311ab3e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:25.187384 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-5236005f-3b84-4f22-aff2-c80af311ab3e None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:41:25.188637 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-5236005f-3b84-4f22-aff2-c80af311ab3e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:41:25.253570 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:25.270480 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 146/296] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:41:25 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 28 00:41:25.286295 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55415076-1ac8-4a6f-9cac-d18b0756a874 None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:25.286885 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55415076-1ac8-4a6f-9cac-d18b0756a874 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:25.287368 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55415076-1ac8-4a6f-9cac-d18b0756a874 None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:25.288911 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 151/297] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:41:25 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:41:25.294592 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a654b3d-64f4-4b61-a3df-a0d126084313 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:25.294880 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a654b3d-64f4-4b61-a3df-a0d126084313 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:25.295153 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a654b3d-64f4-4b61-a3df-a0d126084313 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:25.316105 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-4a654b3d-64f4-4b61-a3df-a0d126084313 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:41:25.317380 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-4a654b3d-64f4-4b61-a3df-a0d126084313 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:41:25.391154 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5460f74c-0922-4ed3-8c4a-27c9a8a71246 None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:25.391478 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5460f74c-0922-4ed3-8c4a-27c9a8a71246 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:25.391725 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5460f74c-0922-4ed3-8c4a-27c9a8a71246 None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:25.394406 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 152/298] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:41:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 28 00:41:25.397121 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 147/299] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:41:25 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 28 00:41:25.399278 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79c2c08a-000b-49a5-a1dd-ac9427d95f7e None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:25.399572 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79c2c08a-000b-49a5-a1dd-ac9427d95f7e None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:25.399849 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79c2c08a-000b-49a5-a1dd-ac9427d95f7e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:25.421237 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-79c2c08a-000b-49a5-a1dd-ac9427d95f7e None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:41:25.422485 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-79c2c08a-000b-49a5-a1dd-ac9427d95f7e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:41:25.451625 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:25.505099 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 153/300] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:41:25 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 28 00:41:25.574606 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a654b3d-64f4-4b61-a3df-a0d126084313 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:25.677672 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a654b3d-64f4-4b61-a3df-a0d126084313 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:25.678255 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79c2c08a-000b-49a5-a1dd-ac9427d95f7e None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:25.698386 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:25.716029 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f93ae271-b472-46c2-b040-16d14acbd088 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:25.716732 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f93ae271-b472-46c2-b040-16d14acbd088 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:25.717041 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f93ae271-b472-46c2-b040-16d14acbd088 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:25.717318 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f93ae271-b472-46c2-b040-16d14acbd088 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:25.727614 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f93ae271-b472-46c2-b040-16d14acbd088 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 28 00:41:25.732691 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-f93ae271-b472-46c2-b040-16d14acbd088 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 28 00:41:25.732691 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:41:25.732691 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:41:25.732691 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:41:25.732691 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:41:25.732691 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:41:25.732691 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:41:25.732691 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:41:25.732691 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:41:25.732691 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:41:25.732691 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:41:25.732691 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:41:25.732691 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 28 00:41:25.732691 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 28 00:41:25.732691 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 28 00:41:25.732691 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 28 00:41:25.733569 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 28 00:41:25.733569 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 28 00:41:25.733569 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 28 00:41:25.733569 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 28 00:41:25.733569 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:41:25.733569 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:41:25.733569 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:41:25.733569 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:41:25.733569 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:41:25.733569 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:41:25.733569 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:41:25.733569 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:41:25.733569 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:41:25.733569 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:41:25.733569 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:41:25.733569 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:41:25.734429 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:41:25.734429 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:41:25.734429 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:41:25.734429 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:41:25.734429 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 28 00:41:25.734429 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 28 00:41:25.734429 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 28 00:41:25.734429 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 28 00:41:25.734429 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 28 00:41:25.734429 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 28 00:41:25.734429 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 28 00:41:25.734429 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:41:25.734429 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 148/301] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Aug 28 00:41:25 2022] GET /identity/v3/roles/member => generated 84 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:41:25.741236 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f93ae271-b472-46c2-b040-16d14acbd088 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:25.786516 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79c2c08a-000b-49a5-a1dd-ac9427d95f7e None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:25.822798 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be3be7b8-b4c8-46b1-a3e5-e530aad403b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:25.823434 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be3be7b8-b4c8-46b1-a3e5-e530aad403b6 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:25.823716 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be3be7b8-b4c8-46b1-a3e5-e530aad403b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:25.823982 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be3be7b8-b4c8-46b1-a3e5-e530aad403b6 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:25.835916 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be3be7b8-b4c8-46b1-a3e5-e530aad403b6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 28 00:41:25.840878 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-be3be7b8-b4c8-46b1-a3e5-e530aad403b6 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 28 00:41:25.840878 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:41:25.840878 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:41:25.840878 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:41:25.840878 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:41:25.840878 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:41:25.840878 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:41:25.840878 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:41:25.840878 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:41:25.840878 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:41:25.840878 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:41:25.840878 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:41:25.840878 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 28 00:41:25.840878 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 28 00:41:25.840878 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 28 00:41:25.840878 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 28 00:41:25.841865 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 28 00:41:25.841865 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 28 00:41:25.841865 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 28 00:41:25.841865 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 28 00:41:25.841865 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:41:25.841865 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:41:25.841865 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:41:25.841865 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:41:25.841865 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:41:25.841865 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:41:25.841865 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:41:25.841865 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:41:25.841865 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:41:25.841865 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:41:25.841865 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:41:25.841865 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:41:25.842665 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:41:25.842665 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:41:25.842665 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:41:25.842665 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:41:25.842665 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 28 00:41:25.842665 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 28 00:41:25.842665 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 28 00:41:25.842665 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 28 00:41:25.842665 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 28 00:41:25.842665 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 28 00:41:25.842665 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 28 00:41:25.842665 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:41:25.847001 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 154/302] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Aug 28 00:41:25 2022] GET /identity/v3/roles/reader => generated 84 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 28 00:41:25.847782 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be3be7b8-b4c8-46b1-a3e5-e530aad403b6 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:25.849314 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f93ae271-b472-46c2-b040-16d14acbd088 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:25.875401 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:25.887305 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b63e158-6719-4287-9acd-892c4c651256 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:25.887915 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b63e158-6719-4287-9acd-892c4c651256 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:25.888252 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b63e158-6719-4287-9acd-892c4c651256 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:25.888527 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b63e158-6719-4287-9acd-892c4c651256 None admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:25.889225 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b63e158-6719-4287-9acd-892c4c651256 None admin] RBAC: Adding query filter params (name=member) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:25.889585 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b63e158-6719-4287-9acd-892c4c651256 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:25.891235 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b63e158-6719-4287-9acd-892c4c651256 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:25.900591 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 149/303] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Aug 28 00:41:25 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:25.901289 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b63e158-6719-4287-9acd-892c4c651256 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:25.944691 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:25.961445 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be3be7b8-b4c8-46b1-a3e5-e530aad403b6 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:25.996407 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a49b13dc-1c84-4a82-9178-16370945b7a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:25.997187 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a49b13dc-1c84-4a82-9178-16370945b7a7 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:25.997526 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a49b13dc-1c84-4a82-9178-16370945b7a7 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:25.997869 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a49b13dc-1c84-4a82-9178-16370945b7a7 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:25.998982 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a49b13dc-1c84-4a82-9178-16370945b7a7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:26.000471 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a49b13dc-1c84-4a82-9178-16370945b7a7 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:26.005113 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 155/304] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:41:25 2022] GET /identity/v3/domains/default => generated 199 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:26.005113 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a49b13dc-1c84-4a82-9178-16370945b7a7 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:26.010677 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b63e158-6719-4287-9acd-892c4c651256 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:26.048339 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffdbe231-e65c-4557-9c97-feeda79077ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:26.048976 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffdbe231-e65c-4557-9c97-feeda79077ca None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:26.049292 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffdbe231-e65c-4557-9c97-feeda79077ca None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:26.049569 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffdbe231-e65c-4557-9c97-feeda79077ca None admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:26.050244 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffdbe231-e65c-4557-9c97-feeda79077ca None admin] RBAC: Adding query filter params (name=reader) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:26.050547 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffdbe231-e65c-4557-9c97-feeda79077ca None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:26.051966 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffdbe231-e65c-4557-9c97-feeda79077ca None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:26.065061 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 150/305] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Aug 28 00:41:25 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:26.065061 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffdbe231-e65c-4557-9c97-feeda79077ca None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:26.117199 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:26.123866 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a49b13dc-1c84-4a82-9178-16370945b7a7 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:26.157401 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d83dfc0-bd11-49e8-b1f9-d0e0d168fa62 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:26.158033 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d83dfc0-bd11-49e8-b1f9-d0e0d168fa62 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:26.158306 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d83dfc0-bd11-49e8-b1f9-d0e0d168fa62 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:26.158867 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d83dfc0-bd11-49e8-b1f9-d0e0d168fa62 None admin] PATH_INFO: `/v3/users/267d82a511ef46d884fe5913687155ca` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:26.159683 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d83dfc0-bd11-49e8-b1f9-d0e0d168fa62 None admin] RBAC: Authorizing `identity:get_user(user_id=267d82a511ef46d884fe5913687155ca)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:26.161524 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d83dfc0-bd11-49e8-b1f9-d0e0d168fa62 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:26.163300 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 156/306] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Aug 28 00:41:26 2022] GET /identity/v3/users/267d82a511ef46d884fe5913687155ca => generated 293 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:26.163876 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d83dfc0-bd11-49e8-b1f9-d0e0d168fa62 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:26.178213 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffdbe231-e65c-4557-9c97-feeda79077ca None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:26.179864 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:26.216497 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-967ea8a1-d2e0-46ed-95f7-fd5fad75046f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:26.217406 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-967ea8a1-d2e0-46ed-95f7-fd5fad75046f None admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:26.217832 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-967ea8a1-d2e0-46ed-95f7-fd5fad75046f None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:26.218250 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-967ea8a1-d2e0-46ed-95f7-fd5fad75046f None admin] PATH_INFO: `/v3/groups` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:26.219277 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-967ea8a1-d2e0-46ed-95f7-fd5fad75046f None admin] RBAC: Authorizing `identity:create_group()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:26.220937 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-967ea8a1-d2e0-46ed-95f7-fd5fad75046f None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:26.241265 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 151/307] 10.222.0.42 () {60 vars in 1134 bytes} [Sun Aug 28 00:41:26 2022] POST /identity/v3/groups => generated 229 bytes in 180 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:41:26.242361 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-967ea8a1-d2e0-46ed-95f7-fd5fad75046f None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:26.275081 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d83dfc0-bd11-49e8-b1f9-d0e0d168fa62 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:26.332036 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab537d78-f975-4f30-a4f1-34df8008c56a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:26.332948 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab537d78-f975-4f30-a4f1-34df8008c56a None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:26.333407 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab537d78-f975-4f30-a4f1-34df8008c56a None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:26.333830 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab537d78-f975-4f30-a4f1-34df8008c56a None admin] PATH_INFO: `/v3/users/6cdb4ded85de4081be852c04ff3557ec` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:26.353190 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab537d78-f975-4f30-a4f1-34df8008c56a None admin] RBAC: Authorizing `identity:get_user(user_id=6cdb4ded85de4081be852c04ff3557ec)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:26.356480 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab537d78-f975-4f30-a4f1-34df8008c56a None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:26.368335 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 157/308] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Aug 28 00:41:26 2022] GET /identity/v3/users/6cdb4ded85de4081be852c04ff3557ec => generated 293 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:26.371059 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab537d78-f975-4f30-a4f1-34df8008c56a None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:26.406881 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:26.408161 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-967ea8a1-d2e0-46ed-95f7-fd5fad75046f None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:26.446715 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2750c3a5-4e5d-423b-9140-d4ec6129dd0a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:26.447369 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2750c3a5-4e5d-423b-9140-d4ec6129dd0a None admin] REQUEST_METHOD: `PUT` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:26.447652 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2750c3a5-4e5d-423b-9140-d4ec6129dd0a None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:26.447925 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2750c3a5-4e5d-423b-9140-d4ec6129dd0a None admin] PATH_INFO: `/v3/system/users/267d82a511ef46d884fe5913687155ca/roles/0487ef7e943e4701915cb8ce0f2899d7` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:26.449390 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2750c3a5-4e5d-423b-9140-d4ec6129dd0a None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=267d82a511ef46d884fe5913687155ca, role_id=0487ef7e943e4701915cb8ce0f2899d7)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:26.450787 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2750c3a5-4e5d-423b-9140-d4ec6129dd0a None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:26.462057 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 152/309] 10.222.0.42 () {58 vars in 1334 bytes} [Sun Aug 28 00:41:26 2022] PUT /identity/v3/system/users/267d82a511ef46d884fe5913687155ca/roles/0487ef7e943e4701915cb8ce0f2899d7 => generated 0 bytes in 220 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:41:26.524891 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab537d78-f975-4f30-a4f1-34df8008c56a None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:26.568982 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-624c4d58-ad87-47a4-8249-c6b6ba4fc99c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:26.569608 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-624c4d58-ad87-47a4-8249-c6b6ba4fc99c None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:26.569887 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-624c4d58-ad87-47a4-8249-c6b6ba4fc99c None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:26.570161 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-624c4d58-ad87-47a4-8249-c6b6ba4fc99c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:26.570961 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-624c4d58-ad87-47a4-8249-c6b6ba4fc99c None admin] RBAC: Adding query filter params (role.id=60823fd2c27d46e98b5d299e86b51918, user.id=6cdb4ded85de4081be852c04ff3557ec, scope.system=all) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:26.571269 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-624c4d58-ad87-47a4-8249-c6b6ba4fc99c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:26.572741 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-624c4d58-ad87-47a4-8249-c6b6ba4fc99c None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:26.579309 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 158/310] 10.222.0.42 () {56 vars in 1308 bytes} [Sun Aug 28 00:41:26 2022] GET /identity/v3/role_assignments?user.id=6cdb4ded85de4081be852c04ff3557ec&scope.system=all&role.id=60823fd2c27d46e98b5d299e86b51918 => generated 226 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 28 00:41:27.096839 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-727d3f40-a2ad-448b-8eb3-26a6ba190bb8 None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:27.097412 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-727d3f40-a2ad-448b-8eb3-26a6ba190bb8 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:27.097412 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-727d3f40-a2ad-448b-8eb3-26a6ba190bb8 None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:27.099075 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 153/311] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:41:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:41:27.104652 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53580abd-d195-4e5d-b886-c05a442817be None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:27.104943 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53580abd-d195-4e5d-b886-c05a442817be None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:27.105253 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53580abd-d195-4e5d-b886-c05a442817be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:27.129773 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-53580abd-d195-4e5d-b886-c05a442817be None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:41:27.130989 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-53580abd-d195-4e5d-b886-c05a442817be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:41:27.225019 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 159/312] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:41:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 28 00:41:27.491847 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-727d3f40-a2ad-448b-8eb3-26a6ba190bb8 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:27.520126 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:27.603488 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-727d3f40-a2ad-448b-8eb3-26a6ba190bb8 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:27.637731 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17a12375-afd6-49d6-8f79-57a3a642fee7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:27.638371 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17a12375-afd6-49d6-8f79-57a3a642fee7 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:27.638645 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17a12375-afd6-49d6-8f79-57a3a642fee7 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:27.638915 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17a12375-afd6-49d6-8f79-57a3a642fee7 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:27.650326 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17a12375-afd6-49d6-8f79-57a3a642fee7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 28 00:41:27.655814 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-17a12375-afd6-49d6-8f79-57a3a642fee7 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 28 00:41:27.655814 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:41:27.655814 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:41:27.655814 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:41:27.655814 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:41:27.655814 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:41:27.655814 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:41:27.655814 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:41:27.655814 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:41:27.655814 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:41:27.655814 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:41:27.655814 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:41:27.655814 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 28 00:41:27.655814 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 28 00:41:27.655814 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 28 00:41:27.655814 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 28 00:41:27.656763 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 28 00:41:27.656763 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 28 00:41:27.656763 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 28 00:41:27.656763 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 28 00:41:27.656763 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:41:27.656763 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:41:27.656763 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:41:27.656763 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:41:27.656763 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:41:27.656763 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:41:27.656763 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:41:27.656763 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:41:27.656763 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:41:27.656763 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:41:27.656763 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:41:27.656763 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:41:27.657615 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:41:27.657615 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:41:27.657615 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:41:27.657615 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:41:27.657615 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 28 00:41:27.657615 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 28 00:41:27.657615 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 28 00:41:27.657615 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 28 00:41:27.657615 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 28 00:41:27.657615 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 28 00:41:27.657615 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 28 00:41:27.657615 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:41:27.657615 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 154/313] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Aug 28 00:41:27 2022] GET /identity/v3/roles/reader => generated 84 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:41:27.663460 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53580abd-d195-4e5d-b886-c05a442817be None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:27.679963 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:27.768485 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53580abd-d195-4e5d-b886-c05a442817be None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:27.802570 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d71272a7-6c74-4a06-8b60-75f203094688 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:27.803181 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d71272a7-6c74-4a06-8b60-75f203094688 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:27.803460 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d71272a7-6c74-4a06-8b60-75f203094688 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:27.803731 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d71272a7-6c74-4a06-8b60-75f203094688 None admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:27.804397 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d71272a7-6c74-4a06-8b60-75f203094688 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:27.804699 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d71272a7-6c74-4a06-8b60-75f203094688 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:27.806125 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d71272a7-6c74-4a06-8b60-75f203094688 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:27.819344 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 160/314] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Aug 28 00:41:27 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:27.820803 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17a12375-afd6-49d6-8f79-57a3a642fee7 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:27.910800 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:27.927527 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17a12375-afd6-49d6-8f79-57a3a642fee7 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:27.973567 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e00cd1b0-4f8b-4625-9890-9481433f60a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:27.974230 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e00cd1b0-4f8b-4625-9890-9481433f60a7 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:27.974508 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e00cd1b0-4f8b-4625-9890-9481433f60a7 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:27.974777 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e00cd1b0-4f8b-4625-9890-9481433f60a7 None admin] PATH_INFO: `/v3/users/6c625c8508504fe7ba38ed46ddab504f` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:27.975980 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e00cd1b0-4f8b-4625-9890-9481433f60a7 None admin] RBAC: Authorizing `identity:get_user(user_id=6c625c8508504fe7ba38ed46ddab504f)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:27.977516 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e00cd1b0-4f8b-4625-9890-9481433f60a7 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:27.979385 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 155/315] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Aug 28 00:41:27 2022] GET /identity/v3/users/6c625c8508504fe7ba38ed46ddab504f => generated 297 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:27.985240 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d71272a7-6c74-4a06-8b60-75f203094688 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:28.085323 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:28.119381 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d71272a7-6c74-4a06-8b60-75f203094688 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:28.155451 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce237aa0-859b-4056-9d17-30a1535f00d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:28.156073 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce237aa0-859b-4056-9d17-30a1535f00d6 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:28.156412 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce237aa0-859b-4056-9d17-30a1535f00d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:28.156686 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce237aa0-859b-4056-9d17-30a1535f00d6 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:28.226604 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce237aa0-859b-4056-9d17-30a1535f00d6 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:28.228071 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce237aa0-859b-4056-9d17-30a1535f00d6 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:28.305373 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:28.334961 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 161/316] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Aug 28 00:41:27 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 350 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 28 00:41:28.340915 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e00cd1b0-4f8b-4625-9890-9481433f60a7 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:28.372294 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-175e30c4-777a-401b-88ed-fa124fa6cce3 None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:28.372615 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-175e30c4-777a-401b-88ed-fa124fa6cce3 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:28.372840 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-175e30c4-777a-401b-88ed-fa124fa6cce3 None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:28.374713 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 162/317] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:41:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:41:28.380993 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c18d142-cc4f-495b-b5f3-282f56bda373 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:28.381316 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c18d142-cc4f-495b-b5f3-282f56bda373 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:28.381535 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c18d142-cc4f-495b-b5f3-282f56bda373 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:28.401813 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-3c18d142-cc4f-495b-b5f3-282f56bda373 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:41:28.403055 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-3c18d142-cc4f-495b-b5f3-282f56bda373 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:41:28.456373 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e00cd1b0-4f8b-4625-9890-9481433f60a7 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:28.481439 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 163/318] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:41:28 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 28 00:41:28.492244 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a0ce613-2c94-4798-84bb-ca6862ec0e48 None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:28.492527 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a0ce613-2c94-4798-84bb-ca6862ec0e48 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:28.492796 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a0ce613-2c94-4798-84bb-ca6862ec0e48 None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:28.494134 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e510cfca-17be-4735-aa09-6c54da7d8ddb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:28.494776 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e510cfca-17be-4735-aa09-6c54da7d8ddb None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:28.495053 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e510cfca-17be-4735-aa09-6c54da7d8ddb None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:28.495323 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e510cfca-17be-4735-aa09-6c54da7d8ddb None admin] PATH_INFO: `/v3/projects/2635535fcc444c74b40bb9c46956c259` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:28.496481 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e510cfca-17be-4735-aa09-6c54da7d8ddb None admin] RBAC: Authorizing `identity:get_project(project_id=2635535fcc444c74b40bb9c46956c259)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:28.498086 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e510cfca-17be-4735-aa09-6c54da7d8ddb None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:28.498287 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 164/319] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:41:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 28 00:41:28.499914 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57c5090d-5f4d-4d12-80c1-4a1fef5e3ac7 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:28.500312 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57c5090d-5f4d-4d12-80c1-4a1fef5e3ac7 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:28.500595 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57c5090d-5f4d-4d12-80c1-4a1fef5e3ac7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:28.500924 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 156/320] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Aug 28 00:41:28 2022] GET /identity/v3/projects/2635535fcc444c74b40bb9c46956c259 => generated 302 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:28.506589 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e510cfca-17be-4735-aa09-6c54da7d8ddb None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:28.527701 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-57c5090d-5f4d-4d12-80c1-4a1fef5e3ac7 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:41:28.528915 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-57c5090d-5f4d-4d12-80c1-4a1fef5e3ac7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:41:28.544953 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:28.612705 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 165/321] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:41:28 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 28 00:41:28.613663 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3ec6581-4c29-4f13-ad78-e42e0d33fccd None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:28.613947 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3ec6581-4c29-4f13-ad78-e42e0d33fccd None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:28.614528 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3ec6581-4c29-4f13-ad78-e42e0d33fccd None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:28.616101 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 166/322] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:41:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:41:28.623427 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a44c5bb7-70a3-4fd3-b7ba-e0c07d0b59b8 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:28.623827 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a44c5bb7-70a3-4fd3-b7ba-e0c07d0b59b8 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:28.624112 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a44c5bb7-70a3-4fd3-b7ba-e0c07d0b59b8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:28.642251 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e510cfca-17be-4735-aa09-6c54da7d8ddb None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:28.649121 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-a44c5bb7-70a3-4fd3-b7ba-e0c07d0b59b8 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:41:28.650902 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-a44c5bb7-70a3-4fd3-b7ba-e0c07d0b59b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:41:28.679046 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b15f3f50-8c52-4397-ae84-5ef927cbb978 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:28.679952 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b15f3f50-8c52-4397-ae84-5ef927cbb978 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:28.680374 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b15f3f50-8c52-4397-ae84-5ef927cbb978 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:28.680784 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b15f3f50-8c52-4397-ae84-5ef927cbb978 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:28.681696 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b15f3f50-8c52-4397-ae84-5ef927cbb978 None admin] RBAC: Adding query filter params (role.id=60823fd2c27d46e98b5d299e86b51918, scope.project.id=2635535fcc444c74b40bb9c46956c259, user.id=6c625c8508504fe7ba38ed46ddab504f) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:28.682141 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b15f3f50-8c52-4397-ae84-5ef927cbb978 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:28.683743 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b15f3f50-8c52-4397-ae84-5ef927cbb978 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:28.689309 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:28.690755 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 157/323] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Aug 28 00:41:28 2022] GET /identity/v3/role_assignments?user.id=6c625c8508504fe7ba38ed46ddab504f&scope.project.id=2635535fcc444c74b40bb9c46956c259&role.id=60823fd2c27d46e98b5d299e86b51918 => generated 608 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 28 00:41:28.738364 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 167/324] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:41:28 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 28 00:41:28.764494 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b15f3f50-8c52-4397-ae84-5ef927cbb978 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:28.864306 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:28.880284 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b15f3f50-8c52-4397-ae84-5ef927cbb978 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:28.901044 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a44c5bb7-70a3-4fd3-b7ba-e0c07d0b59b8 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:28.915909 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fb801d9-958a-4f5e-b694-071d3eef7d64 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:28.916543 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fb801d9-958a-4f5e-b694-071d3eef7d64 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:28.916824 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fb801d9-958a-4f5e-b694-071d3eef7d64 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:28.917124 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fb801d9-958a-4f5e-b694-071d3eef7d64 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:28.918243 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fb801d9-958a-4f5e-b694-071d3eef7d64 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:28.919842 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fb801d9-958a-4f5e-b694-071d3eef7d64 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:28.921890 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 158/325] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:41:28 2022] GET /identity/v3/domains/default => generated 199 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:28.927747 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fb801d9-958a-4f5e-b694-071d3eef7d64 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:29.012022 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a44c5bb7-70a3-4fd3-b7ba-e0c07d0b59b8 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:29.039099 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fb801d9-958a-4f5e-b694-071d3eef7d64 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:29.048256 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e366a6ca-5a64-4eb7-9d0a-af8ff356708f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:29.048886 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e366a6ca-5a64-4eb7-9d0a-af8ff356708f None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:29.049266 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e366a6ca-5a64-4eb7-9d0a-af8ff356708f None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:29.049537 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e366a6ca-5a64-4eb7-9d0a-af8ff356708f None admin] PATH_INFO: `/v3/roles/member` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:29.059477 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e366a6ca-5a64-4eb7-9d0a-af8ff356708f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 28 00:41:29.064394 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-e366a6ca-5a64-4eb7-9d0a-af8ff356708f None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 28 00:41:29.064394 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:41:29.064394 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:41:29.064394 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:41:29.064394 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:41:29.064394 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:41:29.064394 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:41:29.064394 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:41:29.064394 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:41:29.064394 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:41:29.064394 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:41:29.064394 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:41:29.064394 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 28 00:41:29.064394 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 28 00:41:29.064394 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 28 00:41:29.064394 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 28 00:41:29.065443 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 28 00:41:29.065443 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 28 00:41:29.065443 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 28 00:41:29.065443 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 28 00:41:29.065443 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:41:29.065443 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:41:29.065443 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:41:29.065443 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:41:29.065443 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:41:29.065443 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:41:29.065443 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:41:29.065443 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:41:29.065443 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:41:29.065443 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:41:29.065443 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:41:29.065443 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:41:29.066238 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:41:29.066238 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:41:29.066238 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:41:29.066238 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:41:29.066238 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 28 00:41:29.066238 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 28 00:41:29.066238 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 28 00:41:29.066238 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 28 00:41:29.066238 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 28 00:41:29.066238 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 28 00:41:29.066238 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 28 00:41:29.066238 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:41:29.066238 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 168/326] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Aug 28 00:41:28 2022] GET /identity/v3/roles/member => generated 84 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 28 00:41:29.066884 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e366a6ca-5a64-4eb7-9d0a-af8ff356708f None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:29.074788 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88b2e8f5-f1d6-45ac-8a5b-f4ec9f6e24c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:29.075405 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88b2e8f5-f1d6-45ac-8a5b-f4ec9f6e24c9 None admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:29.075688 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88b2e8f5-f1d6-45ac-8a5b-f4ec9f6e24c9 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:29.075961 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88b2e8f5-f1d6-45ac-8a5b-f4ec9f6e24c9 None admin] PATH_INFO: `/v3/groups` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:29.076593 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88b2e8f5-f1d6-45ac-8a5b-f4ec9f6e24c9 None admin] RBAC: Authorizing `identity:create_group()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:29.078148 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88b2e8f5-f1d6-45ac-8a5b-f4ec9f6e24c9 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:29.094167 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 159/327] 10.222.0.42 () {60 vars in 1134 bytes} [Sun Aug 28 00:41:28 2022] POST /identity/v3/groups => generated 226 bytes in 167 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:41:29.095468 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:29.102007 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88b2e8f5-f1d6-45ac-8a5b-f4ec9f6e24c9 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:29.130472 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:29.187179 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e366a6ca-5a64-4eb7-9d0a-af8ff356708f None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:29.224927 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45ee2b92-d5ab-46ee-8aee-159c2b681da2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:29.225600 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45ee2b92-d5ab-46ee-8aee-159c2b681da2 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:29.225873 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45ee2b92-d5ab-46ee-8aee-159c2b681da2 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:29.226140 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45ee2b92-d5ab-46ee-8aee-159c2b681da2 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:29.231642 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88b2e8f5-f1d6-45ac-8a5b-f4ec9f6e24c9 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:29.238616 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45ee2b92-d5ab-46ee-8aee-159c2b681da2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 28 00:41:29.243522 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-45ee2b92-d5ab-46ee-8aee-159c2b681da2 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 28 00:41:29.243522 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:41:29.243522 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:41:29.243522 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:41:29.243522 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:41:29.243522 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:41:29.243522 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:41:29.243522 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:41:29.243522 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:41:29.243522 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:41:29.243522 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:41:29.243522 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:41:29.243522 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 28 00:41:29.243522 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 28 00:41:29.243522 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 28 00:41:29.243522 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 28 00:41:29.244438 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 28 00:41:29.244438 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 28 00:41:29.244438 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 28 00:41:29.244438 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 28 00:41:29.244438 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:41:29.244438 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:41:29.244438 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:41:29.244438 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:41:29.244438 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:41:29.244438 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:41:29.244438 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:41:29.244438 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:41:29.244438 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:41:29.244438 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:41:29.244438 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:41:29.244438 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:41:29.245286 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:41:29.245286 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:41:29.245286 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:41:29.245286 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:41:29.245286 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 28 00:41:29.245286 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 28 00:41:29.245286 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 28 00:41:29.245286 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 28 00:41:29.245286 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 28 00:41:29.245286 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 28 00:41:29.245286 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 28 00:41:29.245286 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:41:29.245286 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 169/328] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Aug 28 00:41:29 2022] GET /identity/v3/roles/reader => generated 84 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:41:29.251025 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45ee2b92-d5ab-46ee-8aee-159c2b681da2 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:29.274764 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-206dd524-0abf-4853-a81d-0f7481f2f524 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:29.275623 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-206dd524-0abf-4853-a81d-0f7481f2f524 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:29.275623 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-206dd524-0abf-4853-a81d-0f7481f2f524 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:29.275899 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-206dd524-0abf-4853-a81d-0f7481f2f524 None admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:29.276562 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-206dd524-0abf-4853-a81d-0f7481f2f524 None admin] RBAC: Adding query filter params (name=member) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:29.277000 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-206dd524-0abf-4853-a81d-0f7481f2f524 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:29.278292 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-206dd524-0abf-4853-a81d-0f7481f2f524 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:29.288721 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 160/329] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Aug 28 00:41:29 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:29.295781 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-206dd524-0abf-4853-a81d-0f7481f2f524 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:29.354642 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:29.385350 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45ee2b92-d5ab-46ee-8aee-159c2b681da2 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:29.397121 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:29.432481 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-206dd524-0abf-4853-a81d-0f7481f2f524 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:29.448950 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ab953b0-30ef-41f7-9e5b-e1ec1ee4b553 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:29.450196 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ab953b0-30ef-41f7-9e5b-e1ec1ee4b553 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:29.450733 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ab953b0-30ef-41f7-9e5b-e1ec1ee4b553 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:29.451241 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ab953b0-30ef-41f7-9e5b-e1ec1ee4b553 None admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:29.452615 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ab953b0-30ef-41f7-9e5b-e1ec1ee4b553 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:29.453194 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ab953b0-30ef-41f7-9e5b-e1ec1ee4b553 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:29.455243 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ab953b0-30ef-41f7-9e5b-e1ec1ee4b553 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:29.468991 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 170/330] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Aug 28 00:41:29 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:29.476368 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ab953b0-30ef-41f7-9e5b-e1ec1ee4b553 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:29.477784 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-832782bf-64b7-4a98-957a-fb6c69bcf84b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:29.478458 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-832782bf-64b7-4a98-957a-fb6c69bcf84b None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:29.478730 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-832782bf-64b7-4a98-957a-fb6c69bcf84b None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:29.478998 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-832782bf-64b7-4a98-957a-fb6c69bcf84b None admin] PATH_INFO: `/v3/users/267d82a511ef46d884fe5913687155ca` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:29.480141 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-832782bf-64b7-4a98-957a-fb6c69bcf84b None admin] RBAC: Authorizing `identity:get_user(user_id=267d82a511ef46d884fe5913687155ca)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:29.481653 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-832782bf-64b7-4a98-957a-fb6c69bcf84b None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:29.483667 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 161/331] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Aug 28 00:41:29 2022] GET /identity/v3/users/267d82a511ef46d884fe5913687155ca => generated 293 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 28 00:41:29.489333 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-832782bf-64b7-4a98-957a-fb6c69bcf84b None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:29.603192 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ab953b0-30ef-41f7-9e5b-e1ec1ee4b553 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:29.616436 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-832782bf-64b7-4a98-957a-fb6c69bcf84b None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:29.642971 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0277a3e9-8055-4526-83c9-31874275981b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:29.643860 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0277a3e9-8055-4526-83c9-31874275981b None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:29.644320 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0277a3e9-8055-4526-83c9-31874275981b None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:29.644761 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0277a3e9-8055-4526-83c9-31874275981b None admin] PATH_INFO: `/v3/users/6cdb4ded85de4081be852c04ff3557ec` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:29.646143 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0277a3e9-8055-4526-83c9-31874275981b None admin] RBAC: Authorizing `identity:get_user(user_id=6cdb4ded85de4081be852c04ff3557ec)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:29.647795 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0277a3e9-8055-4526-83c9-31874275981b None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:29.649989 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 171/332] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Aug 28 00:41:29 2022] GET /identity/v3/users/6cdb4ded85de4081be852c04ff3557ec => generated 293 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:29.657432 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0277a3e9-8055-4526-83c9-31874275981b None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:29.664600 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55e1dee8-2ecc-4e05-8dfe-7ecd174a172c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:29.665332 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55e1dee8-2ecc-4e05-8dfe-7ecd174a172c None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:29.665675 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55e1dee8-2ecc-4e05-8dfe-7ecd174a172c None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:29.666008 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55e1dee8-2ecc-4e05-8dfe-7ecd174a172c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:29.666914 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55e1dee8-2ecc-4e05-8dfe-7ecd174a172c None admin] RBAC: Adding query filter params (role.id=0487ef7e943e4701915cb8ce0f2899d7, user.id=267d82a511ef46d884fe5913687155ca, scope.system=all) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:29.667282 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55e1dee8-2ecc-4e05-8dfe-7ecd174a172c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:29.668816 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55e1dee8-2ecc-4e05-8dfe-7ecd174a172c None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:29.685308 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 162/333] 10.222.0.42 () {56 vars in 1308 bytes} [Sun Aug 28 00:41:29 2022] GET /identity/v3/role_assignments?user.id=267d82a511ef46d884fe5913687155ca&scope.system=all&role.id=0487ef7e943e4701915cb8ce0f2899d7 => generated 510 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 28 00:41:29.686363 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:29.790277 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0277a3e9-8055-4526-83c9-31874275981b None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:29.830539 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b63db97-d5fb-47f4-a5b2-90417307bddd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:29.831597 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b63db97-d5fb-47f4-a5b2-90417307bddd None admin] REQUEST_METHOD: `PUT` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:29.832050 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b63db97-d5fb-47f4-a5b2-90417307bddd None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:29.832555 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b63db97-d5fb-47f4-a5b2-90417307bddd None admin] PATH_INFO: `/v3/system/users/6cdb4ded85de4081be852c04ff3557ec/roles/60823fd2c27d46e98b5d299e86b51918` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:29.834436 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b63db97-d5fb-47f4-a5b2-90417307bddd None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=6cdb4ded85de4081be852c04ff3557ec, role_id=60823fd2c27d46e98b5d299e86b51918)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:29.836176 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b63db97-d5fb-47f4-a5b2-90417307bddd None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:29.853896 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 172/334] 10.222.0.42 () {58 vars in 1334 bytes} [Sun Aug 28 00:41:29 2022] PUT /identity/v3/system/users/6cdb4ded85de4081be852c04ff3557ec/roles/60823fd2c27d46e98b5d299e86b51918 => generated 0 bytes in 197 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:41:31.286620 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc724cb-d900-4a37-8a97-21e71045b511 None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:31.287169 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc724cb-d900-4a37-8a97-21e71045b511 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:31.287169 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc724cb-d900-4a37-8a97-21e71045b511 None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:31.288636 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 163/335] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:41:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:41:31.294118 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-365a474a-14e9-4dde-a639-3248c1f21d9d None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:31.294565 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-365a474a-14e9-4dde-a639-3248c1f21d9d None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:31.294741 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-365a474a-14e9-4dde-a639-3248c1f21d9d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:31.317536 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-365a474a-14e9-4dde-a639-3248c1f21d9d None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:41:31.318723 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-365a474a-14e9-4dde-a639-3248c1f21d9d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:41:31.403795 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 173/336] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:41:31 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 28 00:41:31.639154 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79988ddd-fa42-4df6-af5e-1bf7b418b52a None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:31.639526 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79988ddd-fa42-4df6-af5e-1bf7b418b52a None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:31.639700 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79988ddd-fa42-4df6-af5e-1bf7b418b52a None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:31.641276 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 164/337] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:41:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:41:31.647089 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b586364d-e8ff-4f9f-bdb4-7a93d06dbee1 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:31.647373 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b586364d-e8ff-4f9f-bdb4-7a93d06dbee1 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:31.647641 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b586364d-e8ff-4f9f-bdb4-7a93d06dbee1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:31.673477 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-b586364d-e8ff-4f9f-bdb4-7a93d06dbee1 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:41:31.674600 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-b586364d-e8ff-4f9f-bdb4-7a93d06dbee1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:41:31.676818 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:31.760216 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 174/338] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:41:31 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 28 00:41:31.865608 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79988ddd-fa42-4df6-af5e-1bf7b418b52a None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:31.869237 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:31.994649 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79988ddd-fa42-4df6-af5e-1bf7b418b52a None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:32.035315 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2948bfbb-f65b-43b6-b1b5-c76d4a997c01 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:32.036314 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2948bfbb-f65b-43b6-b1b5-c76d4a997c01 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:32.036820 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2948bfbb-f65b-43b6-b1b5-c76d4a997c01 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:32.037333 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2948bfbb-f65b-43b6-b1b5-c76d4a997c01 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:32.049676 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2948bfbb-f65b-43b6-b1b5-c76d4a997c01 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 28 00:41:32.055460 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-2948bfbb-f65b-43b6-b1b5-c76d4a997c01 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 28 00:41:32.055460 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:41:32.055460 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:41:32.055460 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:41:32.055460 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:41:32.055460 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:41:32.055460 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:41:32.055460 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:41:32.055460 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:41:32.055460 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:41:32.055460 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:41:32.055460 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:41:32.055460 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 28 00:41:32.055460 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 28 00:41:32.055460 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 28 00:41:32.055460 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 28 00:41:32.056443 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 28 00:41:32.056443 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 28 00:41:32.056443 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 28 00:41:32.056443 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 28 00:41:32.056443 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:41:32.056443 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:41:32.056443 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:41:32.056443 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:41:32.056443 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:41:32.056443 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:41:32.056443 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:41:32.056443 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:41:32.056443 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:41:32.056443 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:41:32.056443 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:41:32.056443 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:41:32.057265 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:41:32.057265 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:41:32.057265 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:41:32.057265 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:41:32.057265 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 28 00:41:32.057265 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 28 00:41:32.057265 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 28 00:41:32.057265 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 28 00:41:32.057265 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 28 00:41:32.057265 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 28 00:41:32.057265 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 28 00:41:32.057265 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:41:32.059912 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 165/339] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Aug 28 00:41:31 2022] GET /identity/v3/roles/member => generated 84 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:41:32.069924 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b586364d-e8ff-4f9f-bdb4-7a93d06dbee1 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:32.220821 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b586364d-e8ff-4f9f-bdb4-7a93d06dbee1 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:32.260560 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-320ce62f-550c-4175-ac72-484f305e1f46 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:32.261141 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-320ce62f-550c-4175-ac72-484f305e1f46 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:32.261419 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-320ce62f-550c-4175-ac72-484f305e1f46 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:32.261689 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-320ce62f-550c-4175-ac72-484f305e1f46 None admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:32.262366 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-320ce62f-550c-4175-ac72-484f305e1f46 None admin] RBAC: Adding query filter params (name=member) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:32.262667 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-320ce62f-550c-4175-ac72-484f305e1f46 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:32.264143 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-320ce62f-550c-4175-ac72-484f305e1f46 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:32.270543 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:32.274200 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 175/340] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Aug 28 00:41:32 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:32.280244 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2948bfbb-f65b-43b6-b1b5-c76d4a997c01 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:32.319002 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc8174c-1089-4fc8-ba7b-57172de914d1 None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:32.319842 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc8174c-1089-4fc8-ba7b-57172de914d1 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:32.320280 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc8174c-1089-4fc8-ba7b-57172de914d1 None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:32.322092 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 176/341] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:41:32 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 28 00:41:32.331689 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25728310-7250-4723-bd36-5a9d5e35903b None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:32.332270 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25728310-7250-4723-bd36-5a9d5e35903b None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:32.332737 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25728310-7250-4723-bd36-5a9d5e35903b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:32.363143 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-25728310-7250-4723-bd36-5a9d5e35903b None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:41:32.364681 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-25728310-7250-4723-bd36-5a9d5e35903b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:41:32.388281 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:32.446294 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2948bfbb-f65b-43b6-b1b5-c76d4a997c01 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:32.463406 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 177/342] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:41:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 28 00:41:32.464669 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25728310-7250-4723-bd36-5a9d5e35903b None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:32.528075 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd0e6a98-682e-4ccd-8271-f5d4793999ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:32.528774 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd0e6a98-682e-4ccd-8271-f5d4793999ca None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:32.529288 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd0e6a98-682e-4ccd-8271-f5d4793999ca None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:32.529751 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd0e6a98-682e-4ccd-8271-f5d4793999ca None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:32.584607 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25728310-7250-4723-bd36-5a9d5e35903b None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:32.616283 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:32.624638 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dac37b0-4aed-40fb-8137-f7d7b8324830 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:32.625622 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dac37b0-4aed-40fb-8137-f7d7b8324830 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:32.626048 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dac37b0-4aed-40fb-8137-f7d7b8324830 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:32.626495 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dac37b0-4aed-40fb-8137-f7d7b8324830 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:32.638219 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dac37b0-4aed-40fb-8137-f7d7b8324830 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 28 00:41:32.644012 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-6dac37b0-4aed-40fb-8137-f7d7b8324830 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 28 00:41:32.644012 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:41:32.644012 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:41:32.644012 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:41:32.644012 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:41:32.644012 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:41:32.644012 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:41:32.644012 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:41:32.644012 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:41:32.644012 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:41:32.644012 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:41:32.644012 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:41:32.644012 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 28 00:41:32.644012 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 28 00:41:32.644012 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 28 00:41:32.644012 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 28 00:41:32.644930 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 28 00:41:32.644930 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 28 00:41:32.644930 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 28 00:41:32.644930 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 28 00:41:32.644930 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:41:32.644930 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:41:32.644930 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:41:32.644930 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:41:32.644930 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:41:32.644930 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:41:32.644930 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:41:32.644930 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:41:32.644930 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:41:32.644930 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:41:32.644930 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:41:32.644930 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:41:32.645778 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:41:32.645778 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:41:32.645778 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:41:32.645778 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:41:32.645778 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 28 00:41:32.645778 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 28 00:41:32.645778 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 28 00:41:32.645778 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 28 00:41:32.645778 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 28 00:41:32.645778 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 28 00:41:32.645778 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 28 00:41:32.645778 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:41:32.648214 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 178/343] 10.222.0.42 () {56 vars in 1114 bytes} [Sun Aug 28 00:41:32 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:41:32.651072 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6df558e-819d-4d4d-8f58-d0d3e29da65a None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:32.651581 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6df558e-819d-4d4d-8f58-d0d3e29da65a None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:32.652025 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6df558e-819d-4d4d-8f58-d0d3e29da65a None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:32.653909 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 179/344] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:41:32 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:41:32.654935 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6df558e-819d-4d4d-8f58-d0d3e29da65a None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:32.682000 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd0e6a98-682e-4ccd-8271-f5d4793999ca None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:32.683573 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd0e6a98-682e-4ccd-8271-f5d4793999ca None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:32.766803 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6df558e-819d-4d4d-8f58-d0d3e29da65a None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:32.806679 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08afc083-1091-446c-a5a7-c5d05ef60e0f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:32.807709 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08afc083-1091-446c-a5a7-c5d05ef60e0f None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:32.808207 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08afc083-1091-446c-a5a7-c5d05ef60e0f None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:32.808902 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08afc083-1091-446c-a5a7-c5d05ef60e0f None admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:32.809771 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08afc083-1091-446c-a5a7-c5d05ef60e0f None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:32.810560 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08afc083-1091-446c-a5a7-c5d05ef60e0f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:32.812133 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08afc083-1091-446c-a5a7-c5d05ef60e0f None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:32.824233 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 180/345] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Aug 28 00:41:32 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:32.824918 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:32.827524 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6641267b-cb3d-47c8-a336-33d067656c19 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:32.828377 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6641267b-cb3d-47c8-a336-33d067656c19 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:32.828905 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6641267b-cb3d-47c8-a336-33d067656c19 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:32.852360 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-6641267b-cb3d-47c8-a336-33d067656c19 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:41:32.854023 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-6641267b-cb3d-47c8-a336-33d067656c19 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:41:32.883407 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 166/346] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Aug 28 00:41:32 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 604 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 28 00:41:32.884612 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd0e6a98-682e-4ccd-8271-f5d4793999ca None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:32.915280 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:32.941107 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 181/347] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:41:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 28 00:41:32.942703 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6641267b-cb3d-47c8-a336-33d067656c19 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:33.016180 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd0e6a98-682e-4ccd-8271-f5d4793999ca None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:33.060827 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-750a064d-3213-4104-a592-5961b393e7a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:33.061814 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-750a064d-3213-4104-a592-5961b393e7a7 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:33.062266 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-750a064d-3213-4104-a592-5961b393e7a7 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:33.062775 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-750a064d-3213-4104-a592-5961b393e7a7 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:33.068322 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6641267b-cb3d-47c8-a336-33d067656c19 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:33.077230 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-750a064d-3213-4104-a592-5961b393e7a7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 28 00:41:33.083768 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-750a064d-3213-4104-a592-5961b393e7a7 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 28 00:41:33.083768 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:41:33.083768 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:41:33.083768 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:41:33.083768 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:41:33.083768 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:41:33.083768 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:41:33.083768 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:41:33.083768 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:41:33.083768 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:41:33.083768 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:41:33.083768 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:41:33.083768 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 28 00:41:33.083768 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 28 00:41:33.083768 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 28 00:41:33.083768 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 28 00:41:33.084721 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 28 00:41:33.084721 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 28 00:41:33.084721 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 28 00:41:33.084721 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 28 00:41:33.084721 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:41:33.084721 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:41:33.084721 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:41:33.084721 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:41:33.084721 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:41:33.084721 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:41:33.084721 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:41:33.084721 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:41:33.084721 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:41:33.084721 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:41:33.084721 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:41:33.084721 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:41:33.085565 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:41:33.085565 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:41:33.085565 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:41:33.085565 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:41:33.085565 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 28 00:41:33.085565 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 28 00:41:33.085565 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 28 00:41:33.085565 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 28 00:41:33.085565 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 28 00:41:33.085565 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 28 00:41:33.085565 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 28 00:41:33.085565 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:41:33.088107 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 167/348] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Aug 28 00:41:32 2022] GET /identity/v3/roles/member => generated 84 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:41:33.089336 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-750a064d-3213-4104-a592-5961b393e7a7 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:33.128992 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9555c9bb-3761-4ff8-abb5-2d6a66a3de70 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:33.128992 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9555c9bb-3761-4ff8-abb5-2d6a66a3de70 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:33.128992 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9555c9bb-3761-4ff8-abb5-2d6a66a3de70 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:33.128992 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9555c9bb-3761-4ff8-abb5-2d6a66a3de70 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:33.130057 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:33.224437 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:33.263086 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9555c9bb-3761-4ff8-abb5-2d6a66a3de70 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:33.264599 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9555c9bb-3761-4ff8-abb5-2d6a66a3de70 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:33.278749 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-750a064d-3213-4104-a592-5961b393e7a7 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:33.336847 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ef80575-701d-473f-a019-c61a7bc126a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:33.337831 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef80575-701d-473f-a019-c61a7bc126a2 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:33.338249 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef80575-701d-473f-a019-c61a7bc126a2 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:33.341821 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef80575-701d-473f-a019-c61a7bc126a2 None admin] PATH_INFO: `/v3/projects/39243933b7be4190b983169b99d79165` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:33.343314 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ef80575-701d-473f-a019-c61a7bc126a2 None admin] RBAC: Authorizing `identity:get_project(project_id=39243933b7be4190b983169b99d79165)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:33.345162 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ef80575-701d-473f-a019-c61a7bc126a2 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:33.347137 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 168/349] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Aug 28 00:41:33 2022] GET /identity/v3/projects/39243933b7be4190b983169b99d79165 => generated 298 bytes in 258 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:33.347912 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ef80575-701d-473f-a019-c61a7bc126a2 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:33.401547 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 182/350] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Aug 28 00:41:32 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 460 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 28 00:41:33.403156 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-755da8a2-f73f-482b-944d-c16eb48646cc None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:33.403156 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-755da8a2-f73f-482b-944d-c16eb48646cc None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:33.404349 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-755da8a2-f73f-482b-944d-c16eb48646cc None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:33.405866 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 183/351] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:41:33 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:41:33.406880 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-755da8a2-f73f-482b-944d-c16eb48646cc None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:33.461777 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:33.480092 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ef80575-701d-473f-a019-c61a7bc126a2 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:33.512294 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:33.542259 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-755da8a2-f73f-482b-944d-c16eb48646cc None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:33.544337 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7f06450-98cc-4f32-83c3-6ae265614411 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:33.545289 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7f06450-98cc-4f32-83c3-6ae265614411 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:33.545769 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7f06450-98cc-4f32-83c3-6ae265614411 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:33.546235 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7f06450-98cc-4f32-83c3-6ae265614411 None admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:33.547195 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7f06450-98cc-4f32-83c3-6ae265614411 None admin] RBAC: Adding query filter params (name=member) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:33.547566 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7f06450-98cc-4f32-83c3-6ae265614411 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:33.549043 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7f06450-98cc-4f32-83c3-6ae265614411 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:33.562192 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 169/352] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Aug 28 00:41:33 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:33.563428 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7f06450-98cc-4f32-83c3-6ae265614411 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:33.583259 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18943c67-11d3-4438-9d45-99544baa049b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:33.583896 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18943c67-11d3-4438-9d45-99544baa049b None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:33.584172 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18943c67-11d3-4438-9d45-99544baa049b None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:33.584439 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18943c67-11d3-4438-9d45-99544baa049b None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:33.598589 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18943c67-11d3-4438-9d45-99544baa049b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 28 00:41:33.603463 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-18943c67-11d3-4438-9d45-99544baa049b None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Aug 28 00:41:33.603463 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:41:33.603463 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:41:33.603463 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:41:33.603463 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:41:33.603463 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:41:33.603463 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:41:33.603463 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:41:33.603463 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:41:33.603463 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:41:33.603463 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:41:33.603463 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:41:33.603463 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 28 00:41:33.603463 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 28 00:41:33.603463 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 28 00:41:33.603463 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 28 00:41:33.604414 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 28 00:41:33.604414 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 28 00:41:33.604414 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 28 00:41:33.604414 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 28 00:41:33.604414 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:41:33.604414 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:41:33.604414 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:41:33.604414 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:41:33.604414 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:41:33.604414 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:41:33.604414 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:41:33.604414 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:41:33.604414 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:41:33.604414 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:41:33.604414 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:41:33.604414 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:41:33.605242 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:41:33.605242 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:41:33.605242 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:41:33.605242 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:41:33.605242 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 28 00:41:33.605242 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 28 00:41:33.605242 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 28 00:41:33.605242 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 28 00:41:33.605242 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 28 00:41:33.605242 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 28 00:41:33.605242 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Aug 28 00:41:33.605242 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:41:33.605242 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 184/353] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Aug 28 00:41:33 2022] GET /identity/v3/roles/reader => generated 84 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:41:33.605917 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18943c67-11d3-4438-9d45-99544baa049b None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:33.702695 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7f06450-98cc-4f32-83c3-6ae265614411 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:33.742444 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fd47bd0-c7ee-49b7-b7b1-c54a1b3aee14 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:33.743093 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fd47bd0-c7ee-49b7-b7b1-c54a1b3aee14 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:33.743379 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fd47bd0-c7ee-49b7-b7b1-c54a1b3aee14 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:33.743693 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fd47bd0-c7ee-49b7-b7b1-c54a1b3aee14 None admin] PATH_INFO: `/v3/groups/205c648ef17a42ba92c3a3cbb750dc74` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:33.745833 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18943c67-11d3-4438-9d45-99544baa049b None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:33.750086 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fd47bd0-c7ee-49b7-b7b1-c54a1b3aee14 None admin] RBAC: Authorizing `identity:get_group(group_id=205c648ef17a42ba92c3a3cbb750dc74)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:33.751569 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fd47bd0-c7ee-49b7-b7b1-c54a1b3aee14 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:33.753281 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 170/354] 10.222.0.42 () {56 vars in 1180 bytes} [Sun Aug 28 00:41:33 2022] GET /identity/v3/groups/205c648ef17a42ba92c3a3cbb750dc74 => generated 226 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:33.757531 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a5e9c8c-d03e-4239-b668-e1f73e442d0d None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:33.758442 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a5e9c8c-d03e-4239-b668-e1f73e442d0d None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:33.759083 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a5e9c8c-d03e-4239-b668-e1f73e442d0d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:33.780598 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-5a5e9c8c-d03e-4239-b668-e1f73e442d0d None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:41:33.781992 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-5a5e9c8c-d03e-4239-b668-e1f73e442d0d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:41:33.783784 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b14b00c9-e89c-46d6-b301-75a945f5f499 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:33.784429 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b14b00c9-e89c-46d6-b301-75a945f5f499 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:33.784705 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b14b00c9-e89c-46d6-b301-75a945f5f499 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:33.784994 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b14b00c9-e89c-46d6-b301-75a945f5f499 None admin] PATH_INFO: `/v3/projects/39243933b7be4190b983169b99d79165` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:33.786148 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b14b00c9-e89c-46d6-b301-75a945f5f499 None admin] RBAC: Authorizing `identity:get_project(project_id=39243933b7be4190b983169b99d79165)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:33.787600 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b14b00c9-e89c-46d6-b301-75a945f5f499 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:33.790307 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 185/355] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Aug 28 00:41:33 2022] GET /identity/v3/projects/39243933b7be4190b983169b99d79165 => generated 298 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:33.790502 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b14b00c9-e89c-46d6-b301-75a945f5f499 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:33.796992 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:33.811899 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:33.880990 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 171/356] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:41:33 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 28 00:41:33.881987 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a5e9c8c-d03e-4239-b668-e1f73e442d0d None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:33.918131 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b14b00c9-e89c-46d6-b301-75a945f5f499 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:33.969594 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba1b708f-e3f4-4e4c-9d45-958237e681d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:33.970824 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba1b708f-e3f4-4e4c-9d45-958237e681d0 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:33.971300 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba1b708f-e3f4-4e4c-9d45-958237e681d0 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:33.972625 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba1b708f-e3f4-4e4c-9d45-958237e681d0 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:34.004773 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a5e9c8c-d03e-4239-b668-e1f73e442d0d None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:34.042972 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30765862-55c7-4712-8347-348f75d798b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:34.043590 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30765862-55c7-4712-8347-348f75d798b0 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:34.043866 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30765862-55c7-4712-8347-348f75d798b0 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:34.044152 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30765862-55c7-4712-8347-348f75d798b0 None admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:34.044817 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30765862-55c7-4712-8347-348f75d798b0 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:34.045147 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30765862-55c7-4712-8347-348f75d798b0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:34.046529 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30765862-55c7-4712-8347-348f75d798b0 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:34.052026 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba1b708f-e3f4-4e4c-9d45-958237e681d0 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:34.054841 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba1b708f-e3f4-4e4c-9d45-958237e681d0 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:34.056789 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 172/357] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Aug 28 00:41:33 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:34.057796 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30765862-55c7-4712-8347-348f75d798b0 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:34.071248 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:34.081122 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:34.184717 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30765862-55c7-4712-8347-348f75d798b0 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:34.206289 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 186/358] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Aug 28 00:41:33 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 416 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 28 00:41:34.208099 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba1b708f-e3f4-4e4c-9d45-958237e681d0 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:34.228892 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b8af7fb-1cc4-4dbf-9dce-147106ec4017 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:34.229528 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b8af7fb-1cc4-4dbf-9dce-147106ec4017 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:34.229801 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b8af7fb-1cc4-4dbf-9dce-147106ec4017 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:34.230070 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b8af7fb-1cc4-4dbf-9dce-147106ec4017 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:34.230783 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b8af7fb-1cc4-4dbf-9dce-147106ec4017 None admin] RBAC: Adding query filter params (group.id=205c648ef17a42ba92c3a3cbb750dc74, role.id=0487ef7e943e4701915cb8ce0f2899d7, scope.project.id=39243933b7be4190b983169b99d79165) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:34.231087 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b8af7fb-1cc4-4dbf-9dce-147106ec4017 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:34.232540 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b8af7fb-1cc4-4dbf-9dce-147106ec4017 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:34.238540 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 173/359] 10.222.0.42 () {56 vars in 1376 bytes} [Sun Aug 28 00:41:34 2022] GET /identity/v3/role_assignments?group.id=205c648ef17a42ba92c3a3cbb750dc74&scope.project.id=39243933b7be4190b983169b99d79165&role.id=0487ef7e943e4701915cb8ce0f2899d7 => generated 260 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:34.239665 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b83b6e1c-c4ce-43b2-8549-1dcdd7fd3faa None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:34.240024 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b83b6e1c-c4ce-43b2-8549-1dcdd7fd3faa None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:34.240251 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b83b6e1c-c4ce-43b2-8549-1dcdd7fd3faa None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:34.241549 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 174/360] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:41:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:41:34.242251 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b83b6e1c-c4ce-43b2-8549-1dcdd7fd3faa None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:34.357760 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:34.358993 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba1b708f-e3f4-4e4c-9d45-958237e681d0 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:34.378777 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:34.403420 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b83b6e1c-c4ce-43b2-8549-1dcdd7fd3faa None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:34.411932 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db07cadb-b0dd-45e1-a536-7a84f2fa67e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:34.412877 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db07cadb-b0dd-45e1-a536-7a84f2fa67e1 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:34.412877 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db07cadb-b0dd-45e1-a536-7a84f2fa67e1 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:34.413279 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db07cadb-b0dd-45e1-a536-7a84f2fa67e1 None admin] PATH_INFO: `/v3/groups/205c648ef17a42ba92c3a3cbb750dc74` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:34.414436 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db07cadb-b0dd-45e1-a536-7a84f2fa67e1 None admin] RBAC: Authorizing `identity:get_group(group_id=205c648ef17a42ba92c3a3cbb750dc74)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:34.417666 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db07cadb-b0dd-45e1-a536-7a84f2fa67e1 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:34.417905 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 187/361] 10.222.0.42 () {56 vars in 1180 bytes} [Sun Aug 28 00:41:34 2022] GET /identity/v3/groups/205c648ef17a42ba92c3a3cbb750dc74 => generated 226 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:34.419741 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db07cadb-b0dd-45e1-a536-7a84f2fa67e1 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:34.443957 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6509052a-3248-42c5-a91a-d002a4d17fcb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:34.445415 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6509052a-3248-42c5-a91a-d002a4d17fcb None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:34.445415 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6509052a-3248-42c5-a91a-d002a4d17fcb None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:34.445415 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6509052a-3248-42c5-a91a-d002a4d17fcb None admin] PATH_INFO: `/v3/users/6cdb4ded85de4081be852c04ff3557ec` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:34.447434 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6509052a-3248-42c5-a91a-d002a4d17fcb None admin] RBAC: Authorizing `identity:get_user(user_id=6cdb4ded85de4081be852c04ff3557ec)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:34.448887 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6509052a-3248-42c5-a91a-d002a4d17fcb None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:34.450529 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 175/362] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Aug 28 00:41:34 2022] GET /identity/v3/users/6cdb4ded85de4081be852c04ff3557ec => generated 293 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:34.451399 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6509052a-3248-42c5-a91a-d002a4d17fcb None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:34.568817 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db07cadb-b0dd-45e1-a536-7a84f2fa67e1 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:34.585265 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6509052a-3248-42c5-a91a-d002a4d17fcb None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:34.611455 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bece105d-af62-4269-8fbe-d9b224827866 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:34.612096 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bece105d-af62-4269-8fbe-d9b224827866 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:34.612370 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bece105d-af62-4269-8fbe-d9b224827866 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:34.612636 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bece105d-af62-4269-8fbe-d9b224827866 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:34.627729 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bece105d-af62-4269-8fbe-d9b224827866 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 28 00:41:34.634029 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-bece105d-af62-4269-8fbe-d9b224827866 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 28 00:41:34.634029 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:41:34.634029 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:41:34.634029 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:41:34.634029 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:41:34.634029 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:41:34.634029 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:41:34.634029 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:41:34.634029 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:41:34.634029 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:41:34.634029 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:41:34.634029 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:41:34.634029 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 28 00:41:34.634029 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 28 00:41:34.634029 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 28 00:41:34.634029 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 28 00:41:34.635068 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 28 00:41:34.635068 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 28 00:41:34.635068 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 28 00:41:34.635068 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 28 00:41:34.635068 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:41:34.635068 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:41:34.635068 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:41:34.635068 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:41:34.635068 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:41:34.635068 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:41:34.635068 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:41:34.635068 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:41:34.635068 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:41:34.635068 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:41:34.635068 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:41:34.635068 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:41:34.636423 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:41:34.636423 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:41:34.636423 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:41:34.636423 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:41:34.636423 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 28 00:41:34.636423 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 28 00:41:34.636423 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 28 00:41:34.636423 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 28 00:41:34.636423 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 28 00:41:34.636423 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 28 00:41:34.636423 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 28 00:41:34.636423 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:41:34.636423 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 188/363] 10.222.0.42 () {56 vars in 1114 bytes} [Sun Aug 28 00:41:34 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 217 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:41:34.638640 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f628e78-c773-4e4a-a572-20920b53691d None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:34.639033 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f628e78-c773-4e4a-a572-20920b53691d None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:34.639405 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f628e78-c773-4e4a-a572-20920b53691d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:34.640800 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ab23b8c-27d3-40cd-bd4c-69c251d3a638 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:34.641648 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ab23b8c-27d3-40cd-bd4c-69c251d3a638 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:34.642074 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ab23b8c-27d3-40cd-bd4c-69c251d3a638 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:34.642503 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ab23b8c-27d3-40cd-bd4c-69c251d3a638 None admin] PATH_INFO: `/v3/projects/2635535fcc444c74b40bb9c46956c259` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:34.643915 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ab23b8c-27d3-40cd-bd4c-69c251d3a638 None admin] RBAC: Authorizing `identity:get_project(project_id=2635535fcc444c74b40bb9c46956c259)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:34.645603 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ab23b8c-27d3-40cd-bd4c-69c251d3a638 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:34.647431 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 176/364] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Aug 28 00:41:34 2022] GET /identity/v3/projects/2635535fcc444c74b40bb9c46956c259 => generated 302 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:34.648713 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ab23b8c-27d3-40cd-bd4c-69c251d3a638 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:34.662835 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:34.663514 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-7f628e78-c773-4e4a-a572-20920b53691d None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:41:34.664874 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-7f628e78-c773-4e4a-a572-20920b53691d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:41:34.702163 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:34.750760 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 189/365] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:41:34 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 28 00:41:34.751619 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f628e78-c773-4e4a-a572-20920b53691d None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:34.794125 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ab23b8c-27d3-40cd-bd4c-69c251d3a638 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:34.841925 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-722cd15d-0fb1-45f1-97b1-c82b472c89f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:34.842572 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-722cd15d-0fb1-45f1-97b1-c82b472c89f9 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:34.843290 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-722cd15d-0fb1-45f1-97b1-c82b472c89f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:34.843594 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-722cd15d-0fb1-45f1-97b1-c82b472c89f9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:34.844594 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-722cd15d-0fb1-45f1-97b1-c82b472c89f9 None admin] RBAC: Adding query filter params (group.id=205c648ef17a42ba92c3a3cbb750dc74, role.id=05caf0846a15452b9a6a79fa59c7fa5a, scope.project.id=39243933b7be4190b983169b99d79165) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:34.844899 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-722cd15d-0fb1-45f1-97b1-c82b472c89f9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:34.847018 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-722cd15d-0fb1-45f1-97b1-c82b472c89f9 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:34.854600 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 177/366] 10.222.0.42 () {56 vars in 1376 bytes} [Sun Aug 28 00:41:34 2022] GET /identity/v3/role_assignments?group.id=205c648ef17a42ba92c3a3cbb750dc74&scope.project.id=39243933b7be4190b983169b99d79165&role.id=05caf0846a15452b9a6a79fa59c7fa5a => generated 260 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:34.855607 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-722cd15d-0fb1-45f1-97b1-c82b472c89f9 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:34.865432 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f628e78-c773-4e4a-a572-20920b53691d None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:34.916811 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65359a71-cbd2-49a8-ab52-7e7eb569d74b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:34.917487 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65359a71-cbd2-49a8-ab52-7e7eb569d74b None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:34.917773 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65359a71-cbd2-49a8-ab52-7e7eb569d74b None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:34.918379 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65359a71-cbd2-49a8-ab52-7e7eb569d74b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:34.919679 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65359a71-cbd2-49a8-ab52-7e7eb569d74b None admin] RBAC: Adding query filter params (role.id=60823fd2c27d46e98b5d299e86b51918, user.id=6cdb4ded85de4081be852c04ff3557ec, scope.system=all) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:34.919989 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65359a71-cbd2-49a8-ab52-7e7eb569d74b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:34.921563 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65359a71-cbd2-49a8-ab52-7e7eb569d74b None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:34.939174 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 190/367] 10.222.0.42 () {56 vars in 1308 bytes} [Sun Aug 28 00:41:34 2022] GET /identity/v3/role_assignments?user.id=6cdb4ded85de4081be852c04ff3557ec&scope.system=all&role.id=60823fd2c27d46e98b5d299e86b51918 => generated 510 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:34.940705 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65359a71-cbd2-49a8-ab52-7e7eb569d74b None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:34.945116 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:34.981880 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:35.015991 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-722cd15d-0fb1-45f1-97b1-c82b472c89f9 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:35.091306 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d573a572-38e3-4c36-b8c1-5a10d7304469 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:35.091935 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d573a572-38e3-4c36-b8c1-5a10d7304469 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:35.092216 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d573a572-38e3-4c36-b8c1-5a10d7304469 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:35.092486 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d573a572-38e3-4c36-b8c1-5a10d7304469 None admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:35.093256 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d573a572-38e3-4c36-b8c1-5a10d7304469 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:35.093616 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d573a572-38e3-4c36-b8c1-5a10d7304469 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:35.095114 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d573a572-38e3-4c36-b8c1-5a10d7304469 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:35.108996 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 178/368] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Aug 28 00:41:34 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 254 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:35.114684 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d573a572-38e3-4c36-b8c1-5a10d7304469 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:35.147104 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65359a71-cbd2-49a8-ab52-7e7eb569d74b None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:35.214384 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fd7919f-c84a-4dd9-99a6-30f34ef2e617 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:35.215045 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fd7919f-c84a-4dd9-99a6-30f34ef2e617 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:35.215323 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fd7919f-c84a-4dd9-99a6-30f34ef2e617 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:35.215593 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fd7919f-c84a-4dd9-99a6-30f34ef2e617 None admin] PATH_INFO: `/v3/groups/205c648ef17a42ba92c3a3cbb750dc74` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:35.216727 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fd7919f-c84a-4dd9-99a6-30f34ef2e617 None admin] RBAC: Authorizing `identity:get_group(group_id=205c648ef17a42ba92c3a3cbb750dc74)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:35.218464 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fd7919f-c84a-4dd9-99a6-30f34ef2e617 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:35.220311 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 191/369] 10.222.0.42 () {56 vars in 1180 bytes} [Sun Aug 28 00:41:34 2022] GET /identity/v3/groups/205c648ef17a42ba92c3a3cbb750dc74 => generated 226 bytes in 280 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:35.220751 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fd7919f-c84a-4dd9-99a6-30f34ef2e617 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:35.259089 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d573a572-38e3-4c36-b8c1-5a10d7304469 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:35.297233 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02bb0222-299c-4d9c-b199-91e423dd3c18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:35.298246 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02bb0222-299c-4d9c-b199-91e423dd3c18 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:35.299004 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02bb0222-299c-4d9c-b199-91e423dd3c18 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:35.299510 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02bb0222-299c-4d9c-b199-91e423dd3c18 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:35.302692 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:35.325947 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:35.334130 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fd7919f-c84a-4dd9-99a6-30f34ef2e617 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:35.370504 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-819c0db2-69a6-46d7-9edb-378079872f65 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:35.371643 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-819c0db2-69a6-46d7-9edb-378079872f65 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:35.372264 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-819c0db2-69a6-46d7-9edb-378079872f65 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:35.372895 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-819c0db2-69a6-46d7-9edb-378079872f65 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:35.380738 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02bb0222-299c-4d9c-b199-91e423dd3c18 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:35.384077 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02bb0222-299c-4d9c-b199-91e423dd3c18 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:35.387192 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-819c0db2-69a6-46d7-9edb-378079872f65 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 28 00:41:35.392714 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-819c0db2-69a6-46d7-9edb-378079872f65 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 28 00:41:35.392714 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:41:35.392714 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:41:35.392714 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:41:35.392714 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:41:35.392714 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:41:35.392714 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:41:35.392714 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:41:35.392714 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:41:35.392714 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:41:35.392714 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:41:35.392714 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:41:35.392714 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 28 00:41:35.392714 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 28 00:41:35.392714 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 28 00:41:35.392714 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 28 00:41:35.393790 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 28 00:41:35.393790 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 28 00:41:35.393790 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 28 00:41:35.393790 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 28 00:41:35.393790 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:41:35.393790 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:41:35.393790 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:41:35.393790 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:41:35.393790 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:41:35.393790 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:41:35.393790 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:41:35.393790 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:41:35.393790 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:41:35.393790 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:41:35.393790 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:41:35.393790 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:41:35.394585 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:41:35.394585 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:41:35.394585 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:41:35.394585 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:41:35.394585 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 28 00:41:35.394585 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 28 00:41:35.394585 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 28 00:41:35.394585 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 28 00:41:35.394585 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 28 00:41:35.394585 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 28 00:41:35.394585 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 28 00:41:35.394585 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:41:35.394585 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 192/370] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Aug 28 00:41:35 2022] GET /identity/v3/roles/admin => generated 83 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:41:35.395243 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-819c0db2-69a6-46d7-9edb-378079872f65 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:35.512547 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-819c0db2-69a6-46d7-9edb-378079872f65 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:35.524049 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 179/371] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Aug 28 00:41:35 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 410 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 28 00:41:35.524835 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02bb0222-299c-4d9c-b199-91e423dd3c18 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:35.558255 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96febbe6-f035-441e-b3b1-ee136cc2fbff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:35.559340 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96febbe6-f035-441e-b3b1-ee136cc2fbff None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:35.559808 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96febbe6-f035-441e-b3b1-ee136cc2fbff None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:35.560264 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96febbe6-f035-441e-b3b1-ee136cc2fbff None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:35.561324 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96febbe6-f035-441e-b3b1-ee136cc2fbff None admin] RBAC: Adding query filter params (group.id=205c648ef17a42ba92c3a3cbb750dc74, role.id=0487ef7e943e4701915cb8ce0f2899d7, scope.project.id=2635535fcc444c74b40bb9c46956c259) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:35.561947 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96febbe6-f035-441e-b3b1-ee136cc2fbff None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:35.563641 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96febbe6-f035-441e-b3b1-ee136cc2fbff None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:35.565295 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:35.572455 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 193/372] 10.222.0.42 () {56 vars in 1376 bytes} [Sun Aug 28 00:41:35 2022] GET /identity/v3/role_assignments?group.id=205c648ef17a42ba92c3a3cbb750dc74&scope.project.id=2635535fcc444c74b40bb9c46956c259&role.id=0487ef7e943e4701915cb8ce0f2899d7 => generated 260 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:35.573989 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96febbe6-f035-441e-b3b1-ee136cc2fbff None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:35.609901 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:35.724855 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96febbe6-f035-441e-b3b1-ee136cc2fbff None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:35.744549 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02bb0222-299c-4d9c-b199-91e423dd3c18 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:35.781834 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-480aa3d7-0168-4296-8726-1addfd76beff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:35.784493 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-480aa3d7-0168-4296-8726-1addfd76beff None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:35.787993 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-480aa3d7-0168-4296-8726-1addfd76beff None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:35.788524 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-480aa3d7-0168-4296-8726-1addfd76beff None admin] PATH_INFO: `/v3/projects/2635535fcc444c74b40bb9c46956c259` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:35.790091 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-480aa3d7-0168-4296-8726-1addfd76beff None admin] RBAC: Authorizing `identity:get_project(project_id=2635535fcc444c74b40bb9c46956c259)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:35.791803 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-480aa3d7-0168-4296-8726-1addfd76beff None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:35.795279 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 194/373] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Aug 28 00:41:35 2022] GET /identity/v3/projects/2635535fcc444c74b40bb9c46956c259 => generated 302 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:35.801920 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-480aa3d7-0168-4296-8726-1addfd76beff None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:35.827666 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40f42e85-4824-4e4a-96eb-afb619c0a139 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:35.828346 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f42e85-4824-4e4a-96eb-afb619c0a139 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:35.828629 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f42e85-4824-4e4a-96eb-afb619c0a139 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:35.828902 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f42e85-4824-4e4a-96eb-afb619c0a139 None admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:35.829603 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40f42e85-4824-4e4a-96eb-afb619c0a139 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:35.830231 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40f42e85-4824-4e4a-96eb-afb619c0a139 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:35.831628 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40f42e85-4824-4e4a-96eb-afb619c0a139 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:35.845109 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 180/374] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Aug 28 00:41:35 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 316 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:35.846675 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40f42e85-4824-4e4a-96eb-afb619c0a139 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:35.922675 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:35.952934 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-480aa3d7-0168-4296-8726-1addfd76beff None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:36.001573 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:36.011015 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40f42e85-4824-4e4a-96eb-afb619c0a139 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:36.028114 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5aad8b7-3d2e-4c58-ba68-e655c89f2d71 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:36.028770 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5aad8b7-3d2e-4c58-ba68-e655c89f2d71 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:36.029071 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5aad8b7-3d2e-4c58-ba68-e655c89f2d71 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:36.029341 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5aad8b7-3d2e-4c58-ba68-e655c89f2d71 None admin] PATH_INFO: `/v3/groups/205c648ef17a42ba92c3a3cbb750dc74` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:36.030807 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5aad8b7-3d2e-4c58-ba68-e655c89f2d71 None admin] RBAC: Authorizing `identity:get_group(group_id=205c648ef17a42ba92c3a3cbb750dc74)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:36.032386 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5aad8b7-3d2e-4c58-ba68-e655c89f2d71 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:36.037345 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 195/375] 10.222.0.42 () {56 vars in 1180 bytes} [Sun Aug 28 00:41:35 2022] GET /identity/v3/groups/205c648ef17a42ba92c3a3cbb750dc74 => generated 226 bytes in 234 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:36.041673 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5aad8b7-3d2e-4c58-ba68-e655c89f2d71 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:36.077450 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c7a1b95-629d-4eaf-8a4f-ce259da99eda None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:36.078214 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c7a1b95-629d-4eaf-8a4f-ce259da99eda None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:36.078498 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c7a1b95-629d-4eaf-8a4f-ce259da99eda None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:36.078769 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c7a1b95-629d-4eaf-8a4f-ce259da99eda None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:36.171548 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c7a1b95-629d-4eaf-8a4f-ce259da99eda None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:36.173148 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c7a1b95-629d-4eaf-8a4f-ce259da99eda None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:36.199326 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5aad8b7-3d2e-4c58-ba68-e655c89f2d71 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:36.259445 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63265d20-5bd2-4450-9b3c-2aa16b1dae74 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:36.260065 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63265d20-5bd2-4450-9b3c-2aa16b1dae74 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:36.260340 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63265d20-5bd2-4450-9b3c-2aa16b1dae74 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:36.260610 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63265d20-5bd2-4450-9b3c-2aa16b1dae74 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:36.261368 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63265d20-5bd2-4450-9b3c-2aa16b1dae74 None admin] RBAC: Adding query filter params (group.id=205c648ef17a42ba92c3a3cbb750dc74, role.id=05caf0846a15452b9a6a79fa59c7fa5a, scope.project.id=2635535fcc444c74b40bb9c46956c259) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:36.261671 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63265d20-5bd2-4450-9b3c-2aa16b1dae74 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:36.263104 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63265d20-5bd2-4450-9b3c-2aa16b1dae74 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:36.275412 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 196/376] 10.222.0.42 () {56 vars in 1376 bytes} [Sun Aug 28 00:41:36 2022] GET /identity/v3/role_assignments?group.id=205c648ef17a42ba92c3a3cbb750dc74&scope.project.id=2635535fcc444c74b40bb9c46956c259&role.id=05caf0846a15452b9a6a79fa59c7fa5a => generated 260 bytes in 234 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:36.279582 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 181/377] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Aug 28 00:41:35 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 433 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 28 00:41:36.279991 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:36.286271 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63265d20-5bd2-4450-9b3c-2aa16b1dae74 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:36.325391 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:36.443723 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63265d20-5bd2-4450-9b3c-2aa16b1dae74 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:36.478152 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-081842ee-afc1-43b1-951c-ad1713595a25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:36.478781 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-081842ee-afc1-43b1-951c-ad1713595a25 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:36.479054 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-081842ee-afc1-43b1-951c-ad1713595a25 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:36.479318 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-081842ee-afc1-43b1-951c-ad1713595a25 None admin] PATH_INFO: `/v3/projects/cca4b1c3ff844aae827dd439e67820c7` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:36.480494 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-081842ee-afc1-43b1-951c-ad1713595a25 None admin] RBAC: Authorizing `identity:get_project(project_id=cca4b1c3ff844aae827dd439e67820c7)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:36.481972 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-081842ee-afc1-43b1-951c-ad1713595a25 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:36.483912 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 197/378] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Aug 28 00:41:36 2022] GET /identity/v3/projects/cca4b1c3ff844aae827dd439e67820c7 => generated 344 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:36.489348 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c7a1b95-629d-4eaf-8a4f-ce259da99eda None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:36.616894 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c7a1b95-629d-4eaf-8a4f-ce259da99eda None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:36.659722 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58754f68-68ba-4603-8a00-bff3674ba684 None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:36.660224 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58754f68-68ba-4603-8a00-bff3674ba684 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:36.660299 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58754f68-68ba-4603-8a00-bff3674ba684 None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:36.661629 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 198/379] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:41:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 28 00:41:36.663132 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56915abc-7b8d-4fb5-a9f0-21198c2fa869 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:36.663770 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56915abc-7b8d-4fb5-a9f0-21198c2fa869 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:36.664043 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56915abc-7b8d-4fb5-a9f0-21198c2fa869 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:36.664306 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56915abc-7b8d-4fb5-a9f0-21198c2fa869 None admin] PATH_INFO: `/v3/groups/189372dfaef04a538fb661ce5976dcc8` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:36.671689 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56915abc-7b8d-4fb5-a9f0-21198c2fa869 None admin] RBAC: Authorizing `identity:get_group(group_id=189372dfaef04a538fb661ce5976dcc8)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:36.673163 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56915abc-7b8d-4fb5-a9f0-21198c2fa869 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:36.675071 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 182/380] 10.222.0.42 () {56 vars in 1180 bytes} [Sun Aug 28 00:41:36 2022] GET /identity/v3/groups/189372dfaef04a538fb661ce5976dcc8 => generated 229 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:36.680683 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58754f68-68ba-4603-8a00-bff3674ba684 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:36.682816 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84104987-866d-4d72-9ed8-8eaba0c98b79 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:36.683090 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84104987-866d-4d72-9ed8-8eaba0c98b79 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:36.683354 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84104987-866d-4d72-9ed8-8eaba0c98b79 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:36.711509 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-84104987-866d-4d72-9ed8-8eaba0c98b79 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:41:36.713011 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-84104987-866d-4d72-9ed8-8eaba0c98b79 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:41:36.837205 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:36.837945 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:36.854966 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58754f68-68ba-4603-8a00-bff3674ba684 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:36.860514 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 183/381] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:41:36 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 28 00:41:36.897559 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65206f8b-b90d-4f9a-b005-f99e4b8969fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:36.898177 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65206f8b-b90d-4f9a-b005-f99e4b8969fc None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:36.898458 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65206f8b-b90d-4f9a-b005-f99e4b8969fc None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:36.898735 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65206f8b-b90d-4f9a-b005-f99e4b8969fc None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:36.899468 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65206f8b-b90d-4f9a-b005-f99e4b8969fc None admin] RBAC: Adding query filter params (group.id=189372dfaef04a538fb661ce5976dcc8, role.id=f08d6043261747e3b4763c744eef7dfb, scope.project.id=cca4b1c3ff844aae827dd439e67820c7) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:36.899778 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65206f8b-b90d-4f9a-b005-f99e4b8969fc None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:36.901240 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65206f8b-b90d-4f9a-b005-f99e4b8969fc None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:36.907483 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 199/382] 10.222.0.42 () {56 vars in 1376 bytes} [Sun Aug 28 00:41:36 2022] GET /identity/v3/role_assignments?group.id=189372dfaef04a538fb661ce5976dcc8&scope.project.id=cca4b1c3ff844aae827dd439e67820c7&role.id=f08d6043261747e3b4763c744eef7dfb => generated 260 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:37.318615 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84104987-866d-4d72-9ed8-8eaba0c98b79 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:37.442858 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84104987-866d-4d72-9ed8-8eaba0c98b79 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:37.488581 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81dc0ad4-df44-4191-8f33-c38ffcff0b85 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:37.489264 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81dc0ad4-df44-4191-8f33-c38ffcff0b85 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:37.489547 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81dc0ad4-df44-4191-8f33-c38ffcff0b85 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:37.489820 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81dc0ad4-df44-4191-8f33-c38ffcff0b85 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:37.505347 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81dc0ad4-df44-4191-8f33-c38ffcff0b85 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 28 00:41:37.510261 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-81dc0ad4-df44-4191-8f33-c38ffcff0b85 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 28 00:41:37.510261 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:41:37.510261 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:41:37.510261 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:41:37.510261 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:41:37.510261 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:41:37.510261 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:41:37.510261 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:41:37.510261 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:41:37.510261 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:41:37.510261 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:41:37.510261 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:41:37.510261 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 28 00:41:37.510261 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 28 00:41:37.510261 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 28 00:41:37.510261 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 28 00:41:37.511132 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 28 00:41:37.511132 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 28 00:41:37.511132 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 28 00:41:37.511132 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 28 00:41:37.511132 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:41:37.511132 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:41:37.511132 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:41:37.511132 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:41:37.511132 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:41:37.511132 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:41:37.511132 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:41:37.511132 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:41:37.511132 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:41:37.511132 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:41:37.511132 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:41:37.511132 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:41:37.511908 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:41:37.511908 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:41:37.511908 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:41:37.511908 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:41:37.511908 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 28 00:41:37.511908 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 28 00:41:37.511908 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 28 00:41:37.511908 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 28 00:41:37.511908 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 28 00:41:37.511908 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 28 00:41:37.511908 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 28 00:41:37.511908 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:41:37.511908 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 184/383] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Aug 28 00:41:37 2022] GET /identity/v3/roles/member => generated 84 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:41:37.519092 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65206f8b-b90d-4f9a-b005-f99e4b8969fc None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:37.532149 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a96fd0-36e5-4003-a022-11c9d37494cb None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:37.532451 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a96fd0-36e5-4003-a022-11c9d37494cb None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:37.532700 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a96fd0-36e5-4003-a022-11c9d37494cb None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:37.534418 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 185/384] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:41:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 28 00:41:37.539241 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3da3abb-249a-4587-8dfe-3867e0d6cb42 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:37.539528 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3da3abb-249a-4587-8dfe-3867e0d6cb42 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:37.539803 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3da3abb-249a-4587-8dfe-3867e0d6cb42 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:37.560573 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-f3da3abb-249a-4587-8dfe-3867e0d6cb42 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:41:37.561813 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-f3da3abb-249a-4587-8dfe-3867e0d6cb42 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:41:37.621986 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:37.648248 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65206f8b-b90d-4f9a-b005-f99e4b8969fc None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:37.653747 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 186/385] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:41:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 28 00:41:37.683649 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ac3cbef-1e69-411b-97ac-c96ea4ffe277 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:37.684321 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ac3cbef-1e69-411b-97ac-c96ea4ffe277 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:37.684609 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ac3cbef-1e69-411b-97ac-c96ea4ffe277 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:37.684883 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ac3cbef-1e69-411b-97ac-c96ea4ffe277 None admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:37.685590 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ac3cbef-1e69-411b-97ac-c96ea4ffe277 None admin] RBAC: Adding query filter params (name=member) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:37.685894 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ac3cbef-1e69-411b-97ac-c96ea4ffe277 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:37.687440 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ac3cbef-1e69-411b-97ac-c96ea4ffe277 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:37.696723 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 200/386] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Aug 28 00:41:37 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:37.702336 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3da3abb-249a-4587-8dfe-3867e0d6cb42 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:37.827396 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3da3abb-249a-4587-8dfe-3867e0d6cb42 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:37.867508 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ec687ce-5042-4098-a20c-42e6bba2db42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:37.868271 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ec687ce-5042-4098-a20c-42e6bba2db42 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:37.868426 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ec687ce-5042-4098-a20c-42e6bba2db42 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:37.868685 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ec687ce-5042-4098-a20c-42e6bba2db42 None admin] PATH_INFO: `/v3/groups/205c648ef17a42ba92c3a3cbb750dc74` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:37.873708 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ec687ce-5042-4098-a20c-42e6bba2db42 None admin] RBAC: Authorizing `identity:get_group(group_id=205c648ef17a42ba92c3a3cbb750dc74)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:37.875173 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ec687ce-5042-4098-a20c-42e6bba2db42 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:37.877251 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 187/387] 10.222.0.42 () {56 vars in 1180 bytes} [Sun Aug 28 00:41:37 2022] GET /identity/v3/groups/205c648ef17a42ba92c3a3cbb750dc74 => generated 226 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:37.883465 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ac3cbef-1e69-411b-97ac-c96ea4ffe277 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:37.907916 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:38.008319 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ac3cbef-1e69-411b-97ac-c96ea4ffe277 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:38.038501 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ec687ce-5042-4098-a20c-42e6bba2db42 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:38.044408 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39f7ca5b-c66c-4cdb-ba76-34d83cebe4a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:38.045093 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39f7ca5b-c66c-4cdb-ba76-34d83cebe4a1 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:38.045368 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39f7ca5b-c66c-4cdb-ba76-34d83cebe4a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:38.045643 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39f7ca5b-c66c-4cdb-ba76-34d83cebe4a1 None admin] PATH_INFO: `/v3/projects/39243933b7be4190b983169b99d79165` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:38.050815 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39f7ca5b-c66c-4cdb-ba76-34d83cebe4a1 None admin] RBAC: Authorizing `identity:get_project(project_id=39243933b7be4190b983169b99d79165)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:38.052271 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39f7ca5b-c66c-4cdb-ba76-34d83cebe4a1 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:38.056885 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 201/388] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Aug 28 00:41:37 2022] GET /identity/v3/projects/39243933b7be4190b983169b99d79165 => generated 298 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:38.061251 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39f7ca5b-c66c-4cdb-ba76-34d83cebe4a1 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:38.156892 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:38.198698 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:38.220911 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39f7ca5b-c66c-4cdb-ba76-34d83cebe4a1 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:38.234101 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ec687ce-5042-4098-a20c-42e6bba2db42 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:38.260183 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2abd1975-9d37-463c-9637-89636d7b31f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:38.260850 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2abd1975-9d37-463c-9637-89636d7b31f4 None admin] REQUEST_METHOD: `PUT` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:38.261179 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2abd1975-9d37-463c-9637-89636d7b31f4 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:38.261452 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2abd1975-9d37-463c-9637-89636d7b31f4 None admin] PATH_INFO: `/v3/projects/39243933b7be4190b983169b99d79165/groups/205c648ef17a42ba92c3a3cbb750dc74/roles/0487ef7e943e4701915cb8ce0f2899d7` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:38.263611 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2abd1975-9d37-463c-9637-89636d7b31f4 None admin] RBAC: Authorizing `identity:create_grant(project_id=39243933b7be4190b983169b99d79165, group_id=205c648ef17a42ba92c3a3cbb750dc74, role_id=0487ef7e943e4701915cb8ce0f2899d7)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:38.265044 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2abd1975-9d37-463c-9637-89636d7b31f4 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:38.279144 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-547f4e19-b200-431c-b491-db1ed498cfa4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:38.280119 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-547f4e19-b200-431c-b491-db1ed498cfa4 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:38.280607 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 202/389] 10.222.0.42 () {58 vars in 1442 bytes} [Sun Aug 28 00:41:38 2022] PUT /identity/v3/projects/39243933b7be4190b983169b99d79165/groups/205c648ef17a42ba92c3a3cbb750dc74/roles/0487ef7e943e4701915cb8ce0f2899d7 => generated 0 bytes in 220 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:41:38.280717 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-547f4e19-b200-431c-b491-db1ed498cfa4 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:38.281283 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-547f4e19-b200-431c-b491-db1ed498cfa4 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:38.282175 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fb47fae-bfa8-4c6a-8d6d-9c098392d7f5 None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:38.282456 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fb47fae-bfa8-4c6a-8d6d-9c098392d7f5 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:38.282732 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fb47fae-bfa8-4c6a-8d6d-9c098392d7f5 None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:38.288078 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 203/390] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:41:38 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:41:38.290405 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9bee264-8f66-4862-b0aa-eb68d791df6f None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:38.290691 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9bee264-8f66-4862-b0aa-eb68d791df6f None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:38.290966 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9bee264-8f66-4862-b0aa-eb68d791df6f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:38.308572 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-547f4e19-b200-431c-b491-db1ed498cfa4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 28 00:41:38.315182 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-547f4e19-b200-431c-b491-db1ed498cfa4 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 28 00:41:38.315182 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:41:38.315182 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:41:38.315182 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:41:38.315182 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:41:38.315182 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:41:38.315182 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:41:38.315182 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:41:38.315182 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:41:38.315182 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:41:38.315182 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:41:38.315182 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:41:38.315182 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 28 00:41:38.315182 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 28 00:41:38.315182 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 28 00:41:38.315182 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 28 00:41:38.316150 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 28 00:41:38.316150 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 28 00:41:38.316150 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 28 00:41:38.316150 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 28 00:41:38.316150 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:41:38.316150 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:41:38.316150 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:41:38.316150 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:41:38.316150 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:41:38.316150 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:41:38.316150 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:41:38.316150 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:41:38.316150 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:41:38.316150 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:41:38.316150 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:41:38.316150 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:41:38.316931 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:41:38.316931 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:41:38.316931 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:41:38.316931 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:41:38.316931 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 28 00:41:38.316931 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 28 00:41:38.316931 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 28 00:41:38.316931 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 28 00:41:38.316931 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 28 00:41:38.316931 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 28 00:41:38.316931 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 28 00:41:38.316931 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:41:38.320870 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 188/391] 10.222.0.42 () {56 vars in 1114 bytes} [Sun Aug 28 00:41:38 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 283 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:41:38.326961 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-547f4e19-b200-431c-b491-db1ed498cfa4 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:38.346200 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-e9bee264-8f66-4862-b0aa-eb68d791df6f None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:41:38.347577 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-e9bee264-8f66-4862-b0aa-eb68d791df6f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:41:38.424796 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 204/392] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:41:38 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 28 00:41:38.454581 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-547f4e19-b200-431c-b491-db1ed498cfa4 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:38.465433 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:38.504568 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8301dff5-e9cd-49e7-9f2e-5afe381c0b63 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:38.505405 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8301dff5-e9cd-49e7-9f2e-5afe381c0b63 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:38.505778 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8301dff5-e9cd-49e7-9f2e-5afe381c0b63 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:38.506144 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8301dff5-e9cd-49e7-9f2e-5afe381c0b63 None admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:38.506913 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8301dff5-e9cd-49e7-9f2e-5afe381c0b63 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:38.507386 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8301dff5-e9cd-49e7-9f2e-5afe381c0b63 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:38.508924 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8301dff5-e9cd-49e7-9f2e-5afe381c0b63 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:38.518388 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 189/393] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Aug 28 00:41:38 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:38.524028 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9bee264-8f66-4862-b0aa-eb68d791df6f None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:38.598607 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:38.633689 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9bee264-8f66-4862-b0aa-eb68d791df6f None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:38.667838 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef199f4b-e1ed-4d90-9582-c621dac06a97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:38.668475 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef199f4b-e1ed-4d90-9582-c621dac06a97 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:38.668757 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef199f4b-e1ed-4d90-9582-c621dac06a97 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:38.669060 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef199f4b-e1ed-4d90-9582-c621dac06a97 None admin] PATH_INFO: `/v3/groups/205c648ef17a42ba92c3a3cbb750dc74` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:38.670176 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef199f4b-e1ed-4d90-9582-c621dac06a97 None admin] RBAC: Authorizing `identity:get_group(group_id=205c648ef17a42ba92c3a3cbb750dc74)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:38.671683 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef199f4b-e1ed-4d90-9582-c621dac06a97 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:38.677122 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 205/394] 10.222.0.42 () {56 vars in 1180 bytes} [Sun Aug 28 00:41:38 2022] GET /identity/v3/groups/205c648ef17a42ba92c3a3cbb750dc74 => generated 226 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:38.679334 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8301dff5-e9cd-49e7-9f2e-5afe381c0b63 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:38.729055 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef199f4b-e1ed-4d90-9582-c621dac06a97 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:38.837508 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef199f4b-e1ed-4d90-9582-c621dac06a97 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:38.868018 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8301dff5-e9cd-49e7-9f2e-5afe381c0b63 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:38.878421 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32a37e8d-91df-4fe6-a748-0f9bb4ecdf0c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:38.879554 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32a37e8d-91df-4fe6-a748-0f9bb4ecdf0c None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:38.879837 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32a37e8d-91df-4fe6-a748-0f9bb4ecdf0c None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:38.880385 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32a37e8d-91df-4fe6-a748-0f9bb4ecdf0c None admin] PATH_INFO: `/v3/roles/member` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:38.888950 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:38.894345 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32a37e8d-91df-4fe6-a748-0f9bb4ecdf0c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 28 00:41:38.901273 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-32a37e8d-91df-4fe6-a748-0f9bb4ecdf0c None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 28 00:41:38.901273 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:41:38.901273 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:41:38.901273 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:41:38.901273 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:41:38.901273 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:41:38.901273 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:41:38.901273 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:41:38.901273 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:41:38.901273 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:41:38.901273 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:41:38.901273 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:41:38.901273 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 28 00:41:38.901273 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 28 00:41:38.901273 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 28 00:41:38.901273 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 28 00:41:38.902241 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 28 00:41:38.902241 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 28 00:41:38.902241 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 28 00:41:38.902241 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 28 00:41:38.902241 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:41:38.902241 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:41:38.902241 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:41:38.902241 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:41:38.902241 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:41:38.902241 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:41:38.902241 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:41:38.902241 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:41:38.902241 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:41:38.902241 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:41:38.902241 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:41:38.902241 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:41:38.903041 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:41:38.903041 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:41:38.903041 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:41:38.903041 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:41:38.903041 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 28 00:41:38.903041 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 28 00:41:38.903041 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 28 00:41:38.903041 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 28 00:41:38.903041 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 28 00:41:38.903041 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 28 00:41:38.903041 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 28 00:41:38.903041 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:41:38.903642 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 206/395] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Aug 28 00:41:38 2022] GET /identity/v3/roles/member => generated 84 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 28 00:41:38.905051 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d34647d8-e6ec-4705-940e-c2c79a59a8a1 None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:38.905051 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d34647d8-e6ec-4705-940e-c2c79a59a8a1 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:38.905296 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d34647d8-e6ec-4705-940e-c2c79a59a8a1 None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:38.906800 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 207/396] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:41:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:41:38.908837 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d34647d8-e6ec-4705-940e-c2c79a59a8a1 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:38.915567 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d99bba0-e908-4b9e-981f-470bab0b5394 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:38.916316 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d99bba0-e908-4b9e-981f-470bab0b5394 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:38.916663 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d99bba0-e908-4b9e-981f-470bab0b5394 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:38.917008 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d99bba0-e908-4b9e-981f-470bab0b5394 None admin] PATH_INFO: `/v3/projects/39243933b7be4190b983169b99d79165` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:38.918214 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d99bba0-e908-4b9e-981f-470bab0b5394 None admin] RBAC: Authorizing `identity:get_project(project_id=39243933b7be4190b983169b99d79165)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:38.919901 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d99bba0-e908-4b9e-981f-470bab0b5394 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:38.921997 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 190/397] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Aug 28 00:41:38 2022] GET /identity/v3/projects/39243933b7be4190b983169b99d79165 => generated 298 bytes in 243 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:38.923049 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8365dcf1-91ec-42dd-bd6a-30dfb95dffb7 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:38.923399 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8365dcf1-91ec-42dd-bd6a-30dfb95dffb7 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:38.923728 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8365dcf1-91ec-42dd-bd6a-30dfb95dffb7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:38.945244 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-8365dcf1-91ec-42dd-bd6a-30dfb95dffb7 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:41:38.946631 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-8365dcf1-91ec-42dd-bd6a-30dfb95dffb7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:41:38.978012 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:39.032993 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 191/398] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:41:38 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 28 00:41:39.032993 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8365dcf1-91ec-42dd-bd6a-30dfb95dffb7 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:39.050585 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d34647d8-e6ec-4705-940e-c2c79a59a8a1 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:39.144475 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8365dcf1-91ec-42dd-bd6a-30dfb95dffb7 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:39.169716 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce8769ed-724d-4467-83c5-aa740497cb64 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:39.170342 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce8769ed-724d-4467-83c5-aa740497cb64 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:39.170623 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce8769ed-724d-4467-83c5-aa740497cb64 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:39.170893 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce8769ed-724d-4467-83c5-aa740497cb64 None admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:39.171591 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce8769ed-724d-4467-83c5-aa740497cb64 None admin] RBAC: Adding query filter params (name=member) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:39.171894 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce8769ed-724d-4467-83c5-aa740497cb64 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:39.173304 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce8769ed-724d-4467-83c5-aa740497cb64 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:39.187700 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccbd0779-a886-438d-a401-1e092f857022 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:39.188376 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccbd0779-a886-438d-a401-1e092f857022 None admin] REQUEST_METHOD: `PUT` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:39.188653 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccbd0779-a886-438d-a401-1e092f857022 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:39.188944 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccbd0779-a886-438d-a401-1e092f857022 None admin] PATH_INFO: `/v3/projects/39243933b7be4190b983169b99d79165/groups/205c648ef17a42ba92c3a3cbb750dc74/roles/05caf0846a15452b9a6a79fa59c7fa5a` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:39.191544 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccbd0779-a886-438d-a401-1e092f857022 None admin] RBAC: Authorizing `identity:create_grant(project_id=39243933b7be4190b983169b99d79165, group_id=205c648ef17a42ba92c3a3cbb750dc74, role_id=05caf0846a15452b9a6a79fa59c7fa5a)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:39.193084 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccbd0779-a886-438d-a401-1e092f857022 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:39.195138 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 208/399] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Aug 28 00:41:38 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 286 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 28 00:41:39.201252 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce8769ed-724d-4467-83c5-aa740497cb64 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:39.216782 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 192/400] 10.222.0.42 () {58 vars in 1442 bytes} [Sun Aug 28 00:41:39 2022] PUT /identity/v3/projects/39243933b7be4190b983169b99d79165/groups/205c648ef17a42ba92c3a3cbb750dc74/roles/05caf0846a15452b9a6a79fa59c7fa5a => generated 0 bytes in 188 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:41:39.276652 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d87ec26-b0c6-4e2a-95a0-c84f41c97116 None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:39.277003 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d87ec26-b0c6-4e2a-95a0-c84f41c97116 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:39.277240 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d87ec26-b0c6-4e2a-95a0-c84f41c97116 None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:39.278634 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 193/401] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:41:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 28 00:41:39.284435 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-379035b1-92d8-4f51-a383-467263404717 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:39.284813 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-379035b1-92d8-4f51-a383-467263404717 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:39.285120 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-379035b1-92d8-4f51-a383-467263404717 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:39.298564 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:39.326140 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-379035b1-92d8-4f51-a383-467263404717 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:41:39.327362 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-379035b1-92d8-4f51-a383-467263404717 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:41:39.356809 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce8769ed-724d-4467-83c5-aa740497cb64 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:39.383935 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:39.405744 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0879e217-f37f-4876-9b40-2ead391cd19a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:39.408190 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0879e217-f37f-4876-9b40-2ead391cd19a None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:39.408484 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0879e217-f37f-4876-9b40-2ead391cd19a None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:39.408762 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0879e217-f37f-4876-9b40-2ead391cd19a None admin] PATH_INFO: `/v3/groups/205c648ef17a42ba92c3a3cbb750dc74` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:39.410103 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0879e217-f37f-4876-9b40-2ead391cd19a None admin] RBAC: Authorizing `identity:get_group(group_id=205c648ef17a42ba92c3a3cbb750dc74)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:39.412256 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0879e217-f37f-4876-9b40-2ead391cd19a None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:39.414317 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 209/402] 10.222.0.42 () {56 vars in 1180 bytes} [Sun Aug 28 00:41:39 2022] GET /identity/v3/groups/205c648ef17a42ba92c3a3cbb750dc74 => generated 226 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 28 00:41:39.415744 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0879e217-f37f-4876-9b40-2ead391cd19a None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:39.446256 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 194/403] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:41:39 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 28 00:41:39.447023 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-379035b1-92d8-4f51-a383-467263404717 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:39.619282 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0879e217-f37f-4876-9b40-2ead391cd19a None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:39.636293 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-379035b1-92d8-4f51-a383-467263404717 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:39.684720 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:39.701387 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc1a9186-b2f9-48df-a231-df8c3df052f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:39.702126 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc1a9186-b2f9-48df-a231-df8c3df052f2 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:39.702436 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc1a9186-b2f9-48df-a231-df8c3df052f2 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:39.702793 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc1a9186-b2f9-48df-a231-df8c3df052f2 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:39.715812 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6722cb1a-774f-4cc0-a04e-87057cbe95e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:39.716613 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6722cb1a-774f-4cc0-a04e-87057cbe95e4 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:39.716936 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6722cb1a-774f-4cc0-a04e-87057cbe95e4 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:39.717311 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6722cb1a-774f-4cc0-a04e-87057cbe95e4 None admin] PATH_INFO: `/v3/projects/2635535fcc444c74b40bb9c46956c259` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:39.719900 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6722cb1a-774f-4cc0-a04e-87057cbe95e4 None admin] RBAC: Authorizing `identity:get_project(project_id=2635535fcc444c74b40bb9c46956c259)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:39.721405 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6722cb1a-774f-4cc0-a04e-87057cbe95e4 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:39.730848 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 195/404] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Aug 28 00:41:39 2022] GET /identity/v3/projects/2635535fcc444c74b40bb9c46956c259 => generated 302 bytes in 284 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:39.731743 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc1a9186-b2f9-48df-a231-df8c3df052f2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 28 00:41:39.737182 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-dc1a9186-b2f9-48df-a231-df8c3df052f2 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 28 00:41:39.737182 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:41:39.737182 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:41:39.737182 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:41:39.737182 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:41:39.737182 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:41:39.737182 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:41:39.737182 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:41:39.737182 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:41:39.737182 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:41:39.737182 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:41:39.737182 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:41:39.737182 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 28 00:41:39.737182 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 28 00:41:39.737182 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 28 00:41:39.737182 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 28 00:41:39.738055 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 28 00:41:39.738055 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 28 00:41:39.738055 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 28 00:41:39.738055 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 28 00:41:39.738055 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:41:39.738055 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:41:39.738055 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:41:39.738055 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:41:39.738055 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:41:39.738055 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:41:39.738055 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:41:39.738055 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:41:39.738055 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:41:39.738055 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:41:39.738055 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:41:39.738055 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:41:39.738851 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:41:39.738851 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:41:39.738851 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:41:39.738851 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:41:39.738851 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 28 00:41:39.738851 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 28 00:41:39.738851 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 28 00:41:39.738851 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 28 00:41:39.738851 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 28 00:41:39.738851 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 28 00:41:39.738851 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 28 00:41:39.738851 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:41:39.738851 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 210/405] 10.222.0.42 () {56 vars in 1114 bytes} [Sun Aug 28 00:41:39 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 324 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:41:39.740153 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6722cb1a-774f-4cc0-a04e-87057cbe95e4 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:39.748452 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc1a9186-b2f9-48df-a231-df8c3df052f2 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:39.775375 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:39.853661 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6722cb1a-774f-4cc0-a04e-87057cbe95e4 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:39.878025 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc1a9186-b2f9-48df-a231-df8c3df052f2 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:39.889990 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ac91790-ffeb-4562-9bce-b288e3d5bca1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:39.890778 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ac91790-ffeb-4562-9bce-b288e3d5bca1 None admin] REQUEST_METHOD: `PUT` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:39.891129 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ac91790-ffeb-4562-9bce-b288e3d5bca1 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:39.891475 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ac91790-ffeb-4562-9bce-b288e3d5bca1 None admin] PATH_INFO: `/v3/projects/2635535fcc444c74b40bb9c46956c259/groups/205c648ef17a42ba92c3a3cbb750dc74/roles/0487ef7e943e4701915cb8ce0f2899d7` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:39.893592 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ac91790-ffeb-4562-9bce-b288e3d5bca1 None admin] RBAC: Authorizing `identity:create_grant(project_id=2635535fcc444c74b40bb9c46956c259, group_id=205c648ef17a42ba92c3a3cbb750dc74, role_id=0487ef7e943e4701915cb8ce0f2899d7)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:39.895132 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ac91790-ffeb-4562-9bce-b288e3d5bca1 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:39.913340 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 196/406] 10.222.0.42 () {58 vars in 1442 bytes} [Sun Aug 28 00:41:39 2022] PUT /identity/v3/projects/2635535fcc444c74b40bb9c46956c259/groups/205c648ef17a42ba92c3a3cbb750dc74/roles/0487ef7e943e4701915cb8ce0f2899d7 => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 28 00:41:39.914058 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ac91790-ffeb-4562-9bce-b288e3d5bca1 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:39.918587 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa9ae07c-9c56-49df-b88e-dce18ed4a54c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:39.919414 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa9ae07c-9c56-49df-b88e-dce18ed4a54c None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:39.919761 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa9ae07c-9c56-49df-b88e-dce18ed4a54c None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:39.920101 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa9ae07c-9c56-49df-b88e-dce18ed4a54c None admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:39.921058 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa9ae07c-9c56-49df-b88e-dce18ed4a54c None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:39.922624 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa9ae07c-9c56-49df-b88e-dce18ed4a54c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:39.929027 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa9ae07c-9c56-49df-b88e-dce18ed4a54c None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:39.955483 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 211/407] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Aug 28 00:41:39 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:39.957785 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa9ae07c-9c56-49df-b88e-dce18ed4a54c None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:40.069359 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:40.103355 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:40.140830 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ac91790-ffeb-4562-9bce-b288e3d5bca1 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:40.146312 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa9ae07c-9c56-49df-b88e-dce18ed4a54c None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:40.188032 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18f5ae13-b91a-45b3-8ad4-899b2bd880e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:40.189171 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18f5ae13-b91a-45b3-8ad4-899b2bd880e6 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:40.189630 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18f5ae13-b91a-45b3-8ad4-899b2bd880e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:40.190079 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18f5ae13-b91a-45b3-8ad4-899b2bd880e6 None admin] PATH_INFO: `/v3/groups/205c648ef17a42ba92c3a3cbb750dc74` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:40.191629 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18f5ae13-b91a-45b3-8ad4-899b2bd880e6 None admin] RBAC: Authorizing `identity:get_group(group_id=205c648ef17a42ba92c3a3cbb750dc74)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:40.193767 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18f5ae13-b91a-45b3-8ad4-899b2bd880e6 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:40.195913 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 212/408] 10.222.0.42 () {56 vars in 1180 bytes} [Sun Aug 28 00:41:39 2022] GET /identity/v3/groups/205c648ef17a42ba92c3a3cbb750dc74 => generated 226 bytes in 239 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:40.202128 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18f5ae13-b91a-45b3-8ad4-899b2bd880e6 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:40.249736 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c84d57a-cfce-4d8d-8d5a-c2566a8dad81 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:40.250522 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c84d57a-cfce-4d8d-8d5a-c2566a8dad81 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:40.250867 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c84d57a-cfce-4d8d-8d5a-c2566a8dad81 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:40.251455 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c84d57a-cfce-4d8d-8d5a-c2566a8dad81 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:40.273972 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c84d57a-cfce-4d8d-8d5a-c2566a8dad81 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 28 00:41:40.281636 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-4c84d57a-cfce-4d8d-8d5a-c2566a8dad81 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 28 00:41:40.281636 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:41:40.281636 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:41:40.281636 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:41:40.281636 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:41:40.281636 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:41:40.281636 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:41:40.281636 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:41:40.281636 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:41:40.281636 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:41:40.281636 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:41:40.281636 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:41:40.281636 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 28 00:41:40.281636 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 28 00:41:40.281636 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 28 00:41:40.281636 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 28 00:41:40.284598 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 28 00:41:40.284598 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 28 00:41:40.284598 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 28 00:41:40.284598 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 28 00:41:40.284598 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:41:40.284598 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:41:40.284598 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:41:40.284598 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:41:40.284598 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:41:40.284598 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:41:40.284598 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:41:40.284598 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:41:40.284598 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:41:40.284598 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:41:40.284598 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:41:40.284598 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:41:40.286968 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:41:40.286968 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:41:40.286968 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:41:40.286968 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:41:40.286968 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 28 00:41:40.286968 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 28 00:41:40.286968 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 28 00:41:40.286968 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 28 00:41:40.286968 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 28 00:41:40.286968 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 28 00:41:40.286968 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 28 00:41:40.286968 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:41:40.286968 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 197/409] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Aug 28 00:41:39 2022] GET /identity/v3/roles/admin => generated 83 bytes in 370 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:41:40.292649 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c84d57a-cfce-4d8d-8d5a-c2566a8dad81 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:40.360810 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18f5ae13-b91a-45b3-8ad4-899b2bd880e6 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:40.399621 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:40.429382 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3953a68-c28e-4916-b21f-08f920cd67ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:40.430448 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3953a68-c28e-4916-b21f-08f920cd67ea None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:40.430938 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3953a68-c28e-4916-b21f-08f920cd67ea None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:40.431428 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3953a68-c28e-4916-b21f-08f920cd67ea None admin] PATH_INFO: `/v3/projects/2635535fcc444c74b40bb9c46956c259` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:40.432792 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3953a68-c28e-4916-b21f-08f920cd67ea None admin] RBAC: Authorizing `identity:get_project(project_id=2635535fcc444c74b40bb9c46956c259)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:40.434508 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3953a68-c28e-4916-b21f-08f920cd67ea None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:40.436395 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 213/410] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Aug 28 00:41:40 2022] GET /identity/v3/projects/2635535fcc444c74b40bb9c46956c259 => generated 302 bytes in 235 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:40.442544 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3953a68-c28e-4916-b21f-08f920cd67ea None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:40.514985 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c84d57a-cfce-4d8d-8d5a-c2566a8dad81 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:40.553995 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3953a68-c28e-4916-b21f-08f920cd67ea None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:40.591079 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c9b986f-2e61-42c3-a362-5e6cb461ea7e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:40.592133 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c9b986f-2e61-42c3-a362-5e6cb461ea7e None admin] REQUEST_METHOD: `PUT` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:40.593001 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c9b986f-2e61-42c3-a362-5e6cb461ea7e None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:40.593474 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c9b986f-2e61-42c3-a362-5e6cb461ea7e None admin] PATH_INFO: `/v3/projects/2635535fcc444c74b40bb9c46956c259/groups/205c648ef17a42ba92c3a3cbb750dc74/roles/05caf0846a15452b9a6a79fa59c7fa5a` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:40.595908 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c9b986f-2e61-42c3-a362-5e6cb461ea7e None admin] RBAC: Authorizing `identity:create_grant(project_id=2635535fcc444c74b40bb9c46956c259, group_id=205c648ef17a42ba92c3a3cbb750dc74, role_id=05caf0846a15452b9a6a79fa59c7fa5a)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:40.597646 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c9b986f-2e61-42c3-a362-5e6cb461ea7e None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:40.615023 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 214/411] 10.222.0.42 () {58 vars in 1442 bytes} [Sun Aug 28 00:41:40 2022] PUT /identity/v3/projects/2635535fcc444c74b40bb9c46956c259/groups/205c648ef17a42ba92c3a3cbb750dc74/roles/05caf0846a15452b9a6a79fa59c7fa5a => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 28 00:41:40.630568 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75f99873-1f57-45ef-aa05-dd4c94a668ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:40.631217 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f99873-1f57-45ef-aa05-dd4c94a668ef None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:40.631500 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f99873-1f57-45ef-aa05-dd4c94a668ef None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:40.631769 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f99873-1f57-45ef-aa05-dd4c94a668ef None admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:40.632456 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75f99873-1f57-45ef-aa05-dd4c94a668ef None admin] RBAC: Adding query filter params (name=admin) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:40.632754 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75f99873-1f57-45ef-aa05-dd4c94a668ef None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:40.634218 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75f99873-1f57-45ef-aa05-dd4c94a668ef None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:40.653019 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 198/412] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Aug 28 00:41:40 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 361 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:40.658740 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c9b986f-2e61-42c3-a362-5e6cb461ea7e None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:40.683361 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93c0ab5f-8187-40a5-be39-3051968cc4df None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:40.683703 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93c0ab5f-8187-40a5-be39-3051968cc4df None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:40.683918 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93c0ab5f-8187-40a5-be39-3051968cc4df None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:40.689093 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 199/413] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:41:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 28 00:41:40.698840 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfcfeccb-61ce-4691-b784-f7644b2d9e96 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:40.699153 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfcfeccb-61ce-4691-b784-f7644b2d9e96 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:40.699398 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfcfeccb-61ce-4691-b784-f7644b2d9e96 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:40.730555 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:40.745066 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-cfcfeccb-61ce-4691-b784-f7644b2d9e96 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:41:40.746307 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-cfcfeccb-61ce-4691-b784-f7644b2d9e96 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:41:40.748855 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:40.788181 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c9b986f-2e61-42c3-a362-5e6cb461ea7e None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:40.834882 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56f463ba-2dab-4575-994c-24679c6c3ece None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:40.835535 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f463ba-2dab-4575-994c-24679c6c3ece None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:40.835821 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f463ba-2dab-4575-994c-24679c6c3ece None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:40.836096 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f463ba-2dab-4575-994c-24679c6c3ece None admin] PATH_INFO: `/v3/groups/189372dfaef04a538fb661ce5976dcc8` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:40.841587 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56f463ba-2dab-4575-994c-24679c6c3ece None admin] RBAC: Authorizing `identity:get_group(group_id=189372dfaef04a538fb661ce5976dcc8)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:40.843283 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56f463ba-2dab-4575-994c-24679c6c3ece None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:40.845658 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 215/414] 10.222.0.42 () {56 vars in 1180 bytes} [Sun Aug 28 00:41:40 2022] GET /identity/v3/groups/189372dfaef04a538fb661ce5976dcc8 => generated 229 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:40.851613 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56f463ba-2dab-4575-994c-24679c6c3ece None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:40.894640 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 200/415] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:41:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 197 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 28 00:41:40.990696 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56f463ba-2dab-4575-994c-24679c6c3ece None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:41.016159 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:41.045723 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8357ec58-b09e-4747-bd77-e468bc34c552 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:41.046381 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8357ec58-b09e-4747-bd77-e468bc34c552 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:41.046661 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8357ec58-b09e-4747-bd77-e468bc34c552 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:41.046933 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8357ec58-b09e-4747-bd77-e468bc34c552 None admin] PATH_INFO: `/v3/projects/cca4b1c3ff844aae827dd439e67820c7` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:41.048139 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8357ec58-b09e-4747-bd77-e468bc34c552 None admin] RBAC: Authorizing `identity:get_project(project_id=cca4b1c3ff844aae827dd439e67820c7)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:41.049651 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8357ec58-b09e-4747-bd77-e468bc34c552 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:41.051677 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 216/416] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Aug 28 00:41:40 2022] GET /identity/v3/projects/cca4b1c3ff844aae827dd439e67820c7 => generated 344 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:41.057999 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfcfeccb-61ce-4691-b784-f7644b2d9e96 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:41.183489 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfcfeccb-61ce-4691-b784-f7644b2d9e96 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:41.219988 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c252f76-b568-45e4-8394-24a781c6f616 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:41.220659 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c252f76-b568-45e4-8394-24a781c6f616 None admin] REQUEST_METHOD: `PUT` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:41.220940 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c252f76-b568-45e4-8394-24a781c6f616 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:41.221241 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c252f76-b568-45e4-8394-24a781c6f616 None admin] PATH_INFO: `/v3/projects/cca4b1c3ff844aae827dd439e67820c7/groups/189372dfaef04a538fb661ce5976dcc8/roles/f08d6043261747e3b4763c744eef7dfb` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:41.223265 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c252f76-b568-45e4-8394-24a781c6f616 None admin] RBAC: Authorizing `identity:create_grant(project_id=cca4b1c3ff844aae827dd439e67820c7, group_id=189372dfaef04a538fb661ce5976dcc8, role_id=f08d6043261747e3b4763c744eef7dfb)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:41.224814 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c252f76-b568-45e4-8394-24a781c6f616 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:41.234772 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:41.241022 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 201/417] 10.222.0.42 () {58 vars in 1442 bytes} [Sun Aug 28 00:41:41 2022] PUT /identity/v3/projects/cca4b1c3ff844aae827dd439e67820c7/groups/189372dfaef04a538fb661ce5976dcc8/roles/f08d6043261747e3b4763c744eef7dfb => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:41:41.373430 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8357ec58-b09e-4747-bd77-e468bc34c552 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:41.454345 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-debb1096-b0dd-4892-8f49-727174e3bb2a None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:41.454688 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-debb1096-b0dd-4892-8f49-727174e3bb2a None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:41.455207 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-debb1096-b0dd-4892-8f49-727174e3bb2a None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:41.456564 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 202/418] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:41:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:41:41.462582 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cee0b95a-dd33-4024-8a3d-446de680e6cf None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:41.463158 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cee0b95a-dd33-4024-8a3d-446de680e6cf None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:41.463621 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cee0b95a-dd33-4024-8a3d-446de680e6cf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:41.491012 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-cee0b95a-dd33-4024-8a3d-446de680e6cf None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:41:41.493034 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-cee0b95a-dd33-4024-8a3d-446de680e6cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:41:41.499317 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8357ec58-b09e-4747-bd77-e468bc34c552 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:41.571008 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3072a2cd-7eb6-434e-825c-718102e55495 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:41.571916 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3072a2cd-7eb6-434e-825c-718102e55495 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:41.572425 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3072a2cd-7eb6-434e-825c-718102e55495 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:41.573694 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3072a2cd-7eb6-434e-825c-718102e55495 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:41.595700 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3072a2cd-7eb6-434e-825c-718102e55495 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 28 00:41:41.603707 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-3072a2cd-7eb6-434e-825c-718102e55495 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 28 00:41:41.603707 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:41:41.603707 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:41:41.603707 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:41:41.603707 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:41:41.603707 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:41:41.603707 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:41:41.603707 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:41:41.603707 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:41:41.603707 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:41:41.603707 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:41:41.603707 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:41:41.603707 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 28 00:41:41.603707 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 28 00:41:41.603707 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 28 00:41:41.603707 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 28 00:41:41.604876 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 28 00:41:41.604876 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 28 00:41:41.604876 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 28 00:41:41.604876 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 28 00:41:41.604876 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:41:41.604876 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:41:41.604876 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:41:41.604876 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:41:41.604876 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:41:41.604876 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:41:41.604876 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:41:41.604876 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:41:41.604876 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:41:41.604876 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:41:41.604876 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:41:41.604876 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:41:41.605872 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:41:41.605872 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:41:41.605872 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:41:41.605872 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:41:41.605872 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 28 00:41:41.605872 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 28 00:41:41.605872 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 28 00:41:41.605872 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 28 00:41:41.605872 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 28 00:41:41.605872 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 28 00:41:41.605872 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 28 00:41:41.605872 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:41:41.609409 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 217/419] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Aug 28 00:41:41 2022] GET /identity/v3/roles/member => generated 84 bytes in 237 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:41:41.615755 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3072a2cd-7eb6-434e-825c-718102e55495 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:41.643258 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 203/420] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:41:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 28 00:41:41.677758 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:41.739777 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3072a2cd-7eb6-434e-825c-718102e55495 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:41.774699 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27ec7243-d752-4ecf-8cc8-fdffcc796f96 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:41.775531 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27ec7243-d752-4ecf-8cc8-fdffcc796f96 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:41.775983 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27ec7243-d752-4ecf-8cc8-fdffcc796f96 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:41.776396 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27ec7243-d752-4ecf-8cc8-fdffcc796f96 None admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:41.777242 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27ec7243-d752-4ecf-8cc8-fdffcc796f96 None admin] RBAC: Adding query filter params (name=member) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:41.777696 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27ec7243-d752-4ecf-8cc8-fdffcc796f96 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:41.779220 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27ec7243-d752-4ecf-8cc8-fdffcc796f96 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:41.788317 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 218/421] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Aug 28 00:41:41 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:41.793870 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cee0b95a-dd33-4024-8a3d-446de680e6cf None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:41.902030 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cee0b95a-dd33-4024-8a3d-446de680e6cf None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:41.936254 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:41.946242 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a114d2b-369d-48cd-88c5-38160daed6cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:41.947081 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a114d2b-369d-48cd-88c5-38160daed6cb None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:41.947731 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a114d2b-369d-48cd-88c5-38160daed6cb None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:41.948136 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a114d2b-369d-48cd-88c5-38160daed6cb None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:42.037365 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a114d2b-369d-48cd-88c5-38160daed6cb None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:42.039045 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a114d2b-369d-48cd-88c5-38160daed6cb None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:42.041183 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27ec7243-d752-4ecf-8cc8-fdffcc796f96 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:42.225004 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27ec7243-d752-4ecf-8cc8-fdffcc796f96 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:42.227390 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:42.252273 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 204/422] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Aug 28 00:41:41 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 459 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 28 00:41:42.258122 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a114d2b-369d-48cd-88c5-38160daed6cb None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:42.267500 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-909504ff-6baf-409f-8f44-399bed1d26c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:42.268491 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-909504ff-6baf-409f-8f44-399bed1d26c2 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:42.268775 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-909504ff-6baf-409f-8f44-399bed1d26c2 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:42.269068 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-909504ff-6baf-409f-8f44-399bed1d26c2 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:42.288312 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-909504ff-6baf-409f-8f44-399bed1d26c2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 28 00:41:42.300865 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-909504ff-6baf-409f-8f44-399bed1d26c2 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 28 00:41:42.300865 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:41:42.300865 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:41:42.300865 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:41:42.300865 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:41:42.300865 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:41:42.300865 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:41:42.300865 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:41:42.300865 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:41:42.300865 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:41:42.300865 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:41:42.300865 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:41:42.300865 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 28 00:41:42.300865 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 28 00:41:42.300865 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 28 00:41:42.300865 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 28 00:41:42.301828 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 28 00:41:42.301828 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 28 00:41:42.301828 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 28 00:41:42.301828 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 28 00:41:42.301828 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:41:42.301828 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:41:42.301828 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:41:42.301828 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:41:42.301828 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:41:42.301828 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:41:42.301828 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:41:42.301828 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:41:42.301828 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:41:42.301828 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:41:42.301828 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:41:42.301828 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:41:42.302654 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:41:42.302654 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:41:42.302654 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:41:42.302654 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:41:42.302654 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 28 00:41:42.302654 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 28 00:41:42.302654 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 28 00:41:42.302654 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 28 00:41:42.302654 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 28 00:41:42.302654 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 28 00:41:42.302654 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 28 00:41:42.302654 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:41:42.303741 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 219/423] 10.222.0.42 () {56 vars in 1114 bytes} [Sun Aug 28 00:41:42 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 263 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 28 00:41:42.310075 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-909504ff-6baf-409f-8f44-399bed1d26c2 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:42.311021 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:42.418990 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a114d2b-369d-48cd-88c5-38160daed6cb None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:42.485858 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-909504ff-6baf-409f-8f44-399bed1d26c2 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:42.487227 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02dcca11-903d-42da-a357-69970a59d1dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:42.487883 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02dcca11-903d-42da-a357-69970a59d1dd None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:42.488328 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02dcca11-903d-42da-a357-69970a59d1dd None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:42.488758 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02dcca11-903d-42da-a357-69970a59d1dd None admin] PATH_INFO: `/v3/projects/39243933b7be4190b983169b99d79165` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:42.490184 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02dcca11-903d-42da-a357-69970a59d1dd None admin] RBAC: Authorizing `identity:get_project(project_id=39243933b7be4190b983169b99d79165)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:42.491849 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02dcca11-903d-42da-a357-69970a59d1dd None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:42.493688 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 205/424] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Aug 28 00:41:42 2022] GET /identity/v3/projects/39243933b7be4190b983169b99d79165 => generated 298 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 28 00:41:42.499404 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6a6f256-310a-4e92-a209-339ef3564e96 None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:42.499938 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6a6f256-310a-4e92-a209-339ef3564e96 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:42.500380 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6a6f256-310a-4e92-a209-339ef3564e96 None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:42.501888 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 206/425] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:41:42 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 28 00:41:42.502743 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6a6f256-310a-4e92-a209-339ef3564e96 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:42.525401 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c23a7ba0-b9d3-4262-8021-96aae359b697 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:42.526015 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c23a7ba0-b9d3-4262-8021-96aae359b697 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:42.526296 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c23a7ba0-b9d3-4262-8021-96aae359b697 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:42.526570 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c23a7ba0-b9d3-4262-8021-96aae359b697 None admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:42.527222 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c23a7ba0-b9d3-4262-8021-96aae359b697 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:42.527519 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c23a7ba0-b9d3-4262-8021-96aae359b697 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:42.528885 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c23a7ba0-b9d3-4262-8021-96aae359b697 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:42.539300 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 220/426] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Aug 28 00:41:42 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 230 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:42.540502 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c71d47e5-27f1-4d78-a4de-cc18164ca35c None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:42.540851 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c71d47e5-27f1-4d78-a4de-cc18164ca35c None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:42.541147 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c71d47e5-27f1-4d78-a4de-cc18164ca35c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:42.559954 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-c71d47e5-27f1-4d78-a4de-cc18164ca35c None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:41:42.561338 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-c71d47e5-27f1-4d78-a4de-cc18164ca35c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:41:42.584689 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:42.616081 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:42.625224 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6a6f256-310a-4e92-a209-339ef3564e96 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:42.649010 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 221/427] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:41:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 28 00:41:42.649010 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c71d47e5-27f1-4d78-a4de-cc18164ca35c None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:42.680439 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d5bb447-e695-468c-9cb3-b93da3059250 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:42.681490 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d5bb447-e695-468c-9cb3-b93da3059250 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:42.681991 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d5bb447-e695-468c-9cb3-b93da3059250 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:42.682496 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d5bb447-e695-468c-9cb3-b93da3059250 None admin] PATH_INFO: `/v3/groups/205c648ef17a42ba92c3a3cbb750dc74` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:42.683879 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d5bb447-e695-468c-9cb3-b93da3059250 None admin] RBAC: Authorizing `identity:get_group(group_id=205c648ef17a42ba92c3a3cbb750dc74)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:42.685594 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d5bb447-e695-468c-9cb3-b93da3059250 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:42.687831 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 207/428] 10.222.0.42 () {56 vars in 1180 bytes} [Sun Aug 28 00:41:42 2022] GET /identity/v3/groups/205c648ef17a42ba92c3a3cbb750dc74 => generated 226 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:42.693530 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d5bb447-e695-468c-9cb3-b93da3059250 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:42.756284 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c71d47e5-27f1-4d78-a4de-cc18164ca35c None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:42.803529 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d5bb447-e695-468c-9cb3-b93da3059250 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:42.820995 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c64c5f9-cb8d-4204-af1b-a6ad6851641e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:42.821559 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c64c5f9-cb8d-4204-af1b-a6ad6851641e None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:42.821796 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c64c5f9-cb8d-4204-af1b-a6ad6851641e None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:42.821979 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c64c5f9-cb8d-4204-af1b-a6ad6851641e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:42.843835 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86d929da-5ea3-4783-9f88-6333a3e09ed6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:42.844837 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86d929da-5ea3-4783-9f88-6333a3e09ed6 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:42.845401 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86d929da-5ea3-4783-9f88-6333a3e09ed6 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:42.845871 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86d929da-5ea3-4783-9f88-6333a3e09ed6 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:42.846813 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86d929da-5ea3-4783-9f88-6333a3e09ed6 None admin] RBAC: Adding query filter params (group.id=205c648ef17a42ba92c3a3cbb750dc74, role.id=0487ef7e943e4701915cb8ce0f2899d7, scope.project.id=39243933b7be4190b983169b99d79165) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:42.847330 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86d929da-5ea3-4783-9f88-6333a3e09ed6 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:42.849023 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86d929da-5ea3-4783-9f88-6333a3e09ed6 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:42.856078 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 208/429] 10.222.0.42 () {56 vars in 1376 bytes} [Sun Aug 28 00:41:42 2022] GET /identity/v3/role_assignments?group.id=205c648ef17a42ba92c3a3cbb750dc74&scope.project.id=39243933b7be4190b983169b99d79165&role.id=0487ef7e943e4701915cb8ce0f2899d7 => generated 611 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:42.869337 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:42.919052 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c64c5f9-cb8d-4204-af1b-a6ad6851641e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:42.920490 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aca3a0cd-19b0-46f9-866a-f833eaf3e738 None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:42.921058 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c64c5f9-cb8d-4204-af1b-a6ad6851641e None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:42.921136 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aca3a0cd-19b0-46f9-866a-f833eaf3e738 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:42.921136 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aca3a0cd-19b0-46f9-866a-f833eaf3e738 None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:42.925102 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 209/430] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:41:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 28 00:41:42.932585 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63d4bbe7-042e-457e-ad20-8a11b504ef8f None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:42.932912 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63d4bbe7-042e-457e-ad20-8a11b504ef8f None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:42.933177 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63d4bbe7-042e-457e-ad20-8a11b504ef8f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:42.970664 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-63d4bbe7-042e-457e-ad20-8a11b504ef8f None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:41:42.972122 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-63d4bbe7-042e-457e-ad20-8a11b504ef8f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:41:42.999184 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:43.054679 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 222/431] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Aug 28 00:41:42 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 410 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 28 00:41:43.055547 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c64c5f9-cb8d-4204-af1b-a6ad6851641e None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:43.061745 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 210/432] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:41:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 28 00:41:43.062695 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63d4bbe7-042e-457e-ad20-8a11b504ef8f None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:43.138010 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:43.188032 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c64c5f9-cb8d-4204-af1b-a6ad6851641e None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:43.209226 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63d4bbe7-042e-457e-ad20-8a11b504ef8f None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:43.233338 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71086100-7462-43ec-aab5-1ca1ceceb886 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:43.234151 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71086100-7462-43ec-aab5-1ca1ceceb886 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:43.234504 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71086100-7462-43ec-aab5-1ca1ceceb886 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:43.234843 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71086100-7462-43ec-aab5-1ca1ceceb886 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:43.249363 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71086100-7462-43ec-aab5-1ca1ceceb886 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 28 00:41:43.254832 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06921f98-b0c5-4c14-a9e9-f09837b6a841 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:43.255774 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06921f98-b0c5-4c14-a9e9-f09837b6a841 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:43.256213 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06921f98-b0c5-4c14-a9e9-f09837b6a841 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:43.256685 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06921f98-b0c5-4c14-a9e9-f09837b6a841 None admin] PATH_INFO: `/v3/projects/39243933b7be4190b983169b99d79165` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:43.256787 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-71086100-7462-43ec-aab5-1ca1ceceb886 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Aug 28 00:41:43.256787 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:41:43.256787 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:41:43.256787 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:41:43.256787 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:41:43.256787 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:41:43.256787 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:41:43.256787 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:41:43.256787 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:41:43.256787 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:41:43.256787 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:41:43.256787 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:41:43.256787 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 28 00:41:43.256787 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 28 00:41:43.256787 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 28 00:41:43.256787 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 28 00:41:43.257956 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 28 00:41:43.257956 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 28 00:41:43.257956 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 28 00:41:43.257956 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 28 00:41:43.257956 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:41:43.257956 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:41:43.257956 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:41:43.257956 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:41:43.257956 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:41:43.257956 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:41:43.257956 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:41:43.257956 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:41:43.257956 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:41:43.257956 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:41:43.257956 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:41:43.257956 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:41:43.258969 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:41:43.258969 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:41:43.258969 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:41:43.258969 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:41:43.258969 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 28 00:41:43.258969 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 28 00:41:43.258969 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 28 00:41:43.258969 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 28 00:41:43.258969 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 28 00:41:43.258969 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 28 00:41:43.258969 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Aug 28 00:41:43.258969 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:41:43.260682 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 223/433] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Aug 28 00:41:43 2022] GET /identity/v3/roles/member => generated 84 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:41:43.261656 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06921f98-b0c5-4c14-a9e9-f09837b6a841 None admin] RBAC: Authorizing `identity:get_project(project_id=39243933b7be4190b983169b99d79165)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:43.263362 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06921f98-b0c5-4c14-a9e9-f09837b6a841 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:43.265798 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 211/434] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Aug 28 00:41:43 2022] GET /identity/v3/projects/39243933b7be4190b983169b99d79165 => generated 298 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:43.266669 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71086100-7462-43ec-aab5-1ca1ceceb886 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:43.273221 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06921f98-b0c5-4c14-a9e9-f09837b6a841 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:43.308459 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:43.400374 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71086100-7462-43ec-aab5-1ca1ceceb886 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:43.407826 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06921f98-b0c5-4c14-a9e9-f09837b6a841 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:43.419291 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:43.446417 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e92aa7f-15db-4392-a873-1bb0ce8b5b22 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:43.447290 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e92aa7f-15db-4392-a873-1bb0ce8b5b22 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:43.447740 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e92aa7f-15db-4392-a873-1bb0ce8b5b22 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:43.448167 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e92aa7f-15db-4392-a873-1bb0ce8b5b22 None admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:43.449039 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e92aa7f-15db-4392-a873-1bb0ce8b5b22 None admin] RBAC: Adding query filter params (name=member) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:43.449560 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e92aa7f-15db-4392-a873-1bb0ce8b5b22 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:43.450972 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e92aa7f-15db-4392-a873-1bb0ce8b5b22 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:43.453951 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0189e57e-b5fd-44fb-ab8c-9b8fa5f2fa97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:43.454601 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0189e57e-b5fd-44fb-ab8c-9b8fa5f2fa97 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:43.455184 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0189e57e-b5fd-44fb-ab8c-9b8fa5f2fa97 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:43.455450 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0189e57e-b5fd-44fb-ab8c-9b8fa5f2fa97 None admin] PATH_INFO: `/v3/groups/205c648ef17a42ba92c3a3cbb750dc74` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:43.456600 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0189e57e-b5fd-44fb-ab8c-9b8fa5f2fa97 None admin] RBAC: Authorizing `identity:get_group(group_id=205c648ef17a42ba92c3a3cbb750dc74)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:43.458211 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0189e57e-b5fd-44fb-ab8c-9b8fa5f2fa97 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:43.460121 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 212/435] 10.222.0.42 () {56 vars in 1180 bytes} [Sun Aug 28 00:41:43 2022] GET /identity/v3/groups/205c648ef17a42ba92c3a3cbb750dc74 => generated 226 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:43.460917 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0189e57e-b5fd-44fb-ab8c-9b8fa5f2fa97 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:43.466141 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 224/436] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Aug 28 00:41:43 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 28 00:41:43.468711 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e92aa7f-15db-4392-a873-1bb0ce8b5b22 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:43.582075 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:43.590639 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0189e57e-b5fd-44fb-ab8c-9b8fa5f2fa97 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:43.591335 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e92aa7f-15db-4392-a873-1bb0ce8b5b22 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:43.624902 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbcb599c-5ef4-4af8-a192-58a4c3f7404e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:43.625342 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc9ddc86-810a-4f0a-ab14-1b80c141bc28 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:43.625530 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbcb599c-5ef4-4af8-a192-58a4c3f7404e None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:43.625808 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbcb599c-5ef4-4af8-a192-58a4c3f7404e None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:43.625984 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc9ddc86-810a-4f0a-ab14-1b80c141bc28 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:43.626077 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbcb599c-5ef4-4af8-a192-58a4c3f7404e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:43.626263 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc9ddc86-810a-4f0a-ab14-1b80c141bc28 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:43.626534 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc9ddc86-810a-4f0a-ab14-1b80c141bc28 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:43.626786 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbcb599c-5ef4-4af8-a192-58a4c3f7404e None admin] RBAC: Adding query filter params (group.id=205c648ef17a42ba92c3a3cbb750dc74, role.id=05caf0846a15452b9a6a79fa59c7fa5a, scope.project.id=39243933b7be4190b983169b99d79165) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:43.627090 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbcb599c-5ef4-4af8-a192-58a4c3f7404e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:43.628519 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbcb599c-5ef4-4af8-a192-58a4c3f7404e None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:43.634304 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 225/437] 10.222.0.42 () {56 vars in 1376 bytes} [Sun Aug 28 00:41:43 2022] GET /identity/v3/role_assignments?group.id=205c648ef17a42ba92c3a3cbb750dc74&scope.project.id=39243933b7be4190b983169b99d79165&role.id=05caf0846a15452b9a6a79fa59c7fa5a => generated 611 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:43.635094 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbcb599c-5ef4-4af8-a192-58a4c3f7404e None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:43.637478 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc9ddc86-810a-4f0a-ab14-1b80c141bc28 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 28 00:41:43.643071 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-bc9ddc86-810a-4f0a-ab14-1b80c141bc28 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 28 00:41:43.643071 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:41:43.643071 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:41:43.643071 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:41:43.643071 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:41:43.643071 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:41:43.643071 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:41:43.643071 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:41:43.643071 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:41:43.643071 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:41:43.643071 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:41:43.643071 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:41:43.643071 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 28 00:41:43.643071 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 28 00:41:43.643071 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 28 00:41:43.643071 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 28 00:41:43.644067 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 28 00:41:43.644067 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 28 00:41:43.644067 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 28 00:41:43.644067 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 28 00:41:43.644067 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:41:43.644067 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:41:43.644067 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:41:43.644067 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:41:43.644067 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:41:43.644067 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:41:43.644067 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:41:43.644067 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:41:43.644067 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:41:43.644067 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:41:43.644067 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:41:43.644067 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:41:43.644980 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:41:43.644980 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:41:43.644980 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:41:43.644980 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:41:43.644980 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 28 00:41:43.644980 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 28 00:41:43.644980 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 28 00:41:43.644980 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 28 00:41:43.644980 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 28 00:41:43.644980 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 28 00:41:43.644980 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 28 00:41:43.644980 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:41:43.644980 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 213/438] 10.222.0.42 () {56 vars in 1114 bytes} [Sun Aug 28 00:41:43 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:41:43.646497 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22817f28-c66b-488f-912f-836ddbff4908 None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:43.646794 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22817f28-c66b-488f-912f-836ddbff4908 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:43.647074 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22817f28-c66b-488f-912f-836ddbff4908 None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:43.648477 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 214/439] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:41:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:41:43.653589 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22817f28-c66b-488f-912f-836ddbff4908 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:43.689997 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:43.760706 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbcb599c-5ef4-4af8-a192-58a4c3f7404e None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:43.772052 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22817f28-c66b-488f-912f-836ddbff4908 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:43.800758 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e82593b-4041-44bb-8113-4e58947bd6d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:43.801414 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e82593b-4041-44bb-8113-4e58947bd6d2 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:43.801692 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e82593b-4041-44bb-8113-4e58947bd6d2 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:43.801964 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e82593b-4041-44bb-8113-4e58947bd6d2 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:43.820673 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d24df3ec-1e1e-48a3-8d3d-fdc91a5b04f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:43.821318 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d24df3ec-1e1e-48a3-8d3d-fdc91a5b04f7 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:43.822110 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d24df3ec-1e1e-48a3-8d3d-fdc91a5b04f7 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:43.822110 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d24df3ec-1e1e-48a3-8d3d-fdc91a5b04f7 None admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:43.822504 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d24df3ec-1e1e-48a3-8d3d-fdc91a5b04f7 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:43.822798 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d24df3ec-1e1e-48a3-8d3d-fdc91a5b04f7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:43.824181 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d24df3ec-1e1e-48a3-8d3d-fdc91a5b04f7 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:43.833797 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 215/440] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Aug 28 00:41:43 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:43.838532 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22122121-0d7a-4720-a484-6a338b73e7eb None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:43.838532 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22122121-0d7a-4720-a484-6a338b73e7eb None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:43.838532 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22122121-0d7a-4720-a484-6a338b73e7eb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:43.854552 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-22122121-0d7a-4720-a484-6a338b73e7eb None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:41:43.855936 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-22122121-0d7a-4720-a484-6a338b73e7eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:41:43.868892 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:43.887982 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e82593b-4041-44bb-8113-4e58947bd6d2 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:43.890405 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e82593b-4041-44bb-8113-4e58947bd6d2 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:43.938544 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 216/441] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:41:43 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 28 00:41:43.939994 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22122121-0d7a-4720-a484-6a338b73e7eb None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:43.946958 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:44.031588 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 226/442] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Aug 28 00:41:43 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 397 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 28 00:41:44.037862 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e82593b-4041-44bb-8113-4e58947bd6d2 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:44.081013 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22122121-0d7a-4720-a484-6a338b73e7eb None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:44.126175 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-261313d6-f140-4b14-a47c-638882ab8765 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:44.126796 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-261313d6-f140-4b14-a47c-638882ab8765 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:44.127076 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-261313d6-f140-4b14-a47c-638882ab8765 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:44.127349 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-261313d6-f140-4b14-a47c-638882ab8765 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:44.143693 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:44.154401 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e82593b-4041-44bb-8113-4e58947bd6d2 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:44.192025 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1821224-dedd-43e2-b525-dace6d3cdb5b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:44.192661 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1821224-dedd-43e2-b525-dace6d3cdb5b None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:44.192940 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1821224-dedd-43e2-b525-dace6d3cdb5b None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:44.193246 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1821224-dedd-43e2-b525-dace6d3cdb5b None admin] PATH_INFO: `/v3/projects/2635535fcc444c74b40bb9c46956c259` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:44.194490 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1821224-dedd-43e2-b525-dace6d3cdb5b None admin] RBAC: Authorizing `identity:get_project(project_id=2635535fcc444c74b40bb9c46956c259)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:44.196071 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1821224-dedd-43e2-b525-dace6d3cdb5b None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:44.197372 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-261313d6-f140-4b14-a47c-638882ab8765 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:44.197917 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 227/443] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Aug 28 00:41:44 2022] GET /identity/v3/projects/2635535fcc444c74b40bb9c46956c259 => generated 302 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:44.198850 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-261313d6-f140-4b14-a47c-638882ab8765 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:44.203522 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1821224-dedd-43e2-b525-dace6d3cdb5b None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:44.216219 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:44.304151 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1821224-dedd-43e2-b525-dace6d3cdb5b None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:44.311336 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 217/444] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Aug 28 00:41:43 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 372 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 28 00:41:44.312423 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-261313d6-f140-4b14-a47c-638882ab8765 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:44.342365 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3ccc89c-2cdb-4091-aea8-4cda40818ca6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:44.342992 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3ccc89c-2cdb-4091-aea8-4cda40818ca6 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:44.343267 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3ccc89c-2cdb-4091-aea8-4cda40818ca6 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:44.343727 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3ccc89c-2cdb-4091-aea8-4cda40818ca6 None admin] PATH_INFO: `/v3/groups/205c648ef17a42ba92c3a3cbb750dc74` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:44.345116 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3ccc89c-2cdb-4091-aea8-4cda40818ca6 None admin] RBAC: Authorizing `identity:get_group(group_id=205c648ef17a42ba92c3a3cbb750dc74)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:44.346560 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3ccc89c-2cdb-4091-aea8-4cda40818ca6 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:44.348126 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 228/445] 10.222.0.42 () {56 vars in 1180 bytes} [Sun Aug 28 00:41:44 2022] GET /identity/v3/groups/205c648ef17a42ba92c3a3cbb750dc74 => generated 226 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 28 00:41:44.348897 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3ccc89c-2cdb-4091-aea8-4cda40818ca6 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:44.366853 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:44.420202 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-261313d6-f140-4b14-a47c-638882ab8765 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:44.428480 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:44.451534 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3ccc89c-2cdb-4091-aea8-4cda40818ca6 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:44.455983 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b528670-abac-4ec6-8fea-60eb5d5d7e6f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:44.456620 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b528670-abac-4ec6-8fea-60eb5d5d7e6f None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:44.456895 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b528670-abac-4ec6-8fea-60eb5d5d7e6f None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:44.457192 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b528670-abac-4ec6-8fea-60eb5d5d7e6f None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:44.467886 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b528670-abac-4ec6-8fea-60eb5d5d7e6f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 28 00:41:44.472835 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-8b528670-abac-4ec6-8fea-60eb5d5d7e6f None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 28 00:41:44.472835 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:41:44.472835 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:41:44.472835 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:41:44.472835 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:41:44.472835 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:41:44.472835 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:41:44.472835 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:41:44.472835 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:41:44.472835 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:41:44.472835 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:41:44.472835 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:41:44.472835 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 28 00:41:44.472835 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 28 00:41:44.472835 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 28 00:41:44.472835 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 28 00:41:44.473779 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 28 00:41:44.473779 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 28 00:41:44.473779 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 28 00:41:44.473779 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 28 00:41:44.473779 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:41:44.473779 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:41:44.473779 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:41:44.473779 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:41:44.473779 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:41:44.473779 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:41:44.473779 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:41:44.473779 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:41:44.473779 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:41:44.473779 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:41:44.473779 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:41:44.473779 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:41:44.476925 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:41:44.476925 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:41:44.476925 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:41:44.476925 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:41:44.476925 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 28 00:41:44.476925 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 28 00:41:44.476925 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 28 00:41:44.476925 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 28 00:41:44.476925 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 28 00:41:44.476925 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 28 00:41:44.476925 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 28 00:41:44.476925 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:41:44.476925 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 218/446] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Aug 28 00:41:44 2022] GET /identity/v3/roles/admin => generated 83 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:41:44.476925 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b528670-abac-4ec6-8fea-60eb5d5d7e6f None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:44.495916 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67db52a7-39c2-47f3-a240-36dac318ba0b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:44.496568 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67db52a7-39c2-47f3-a240-36dac318ba0b None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:44.496843 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67db52a7-39c2-47f3-a240-36dac318ba0b None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:44.497135 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67db52a7-39c2-47f3-a240-36dac318ba0b None admin] PATH_INFO: `/v3/projects/2635535fcc444c74b40bb9c46956c259` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:44.498345 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67db52a7-39c2-47f3-a240-36dac318ba0b None admin] RBAC: Authorizing `identity:get_project(project_id=2635535fcc444c74b40bb9c46956c259)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:44.499949 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67db52a7-39c2-47f3-a240-36dac318ba0b None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:44.501677 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 229/447] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Aug 28 00:41:44 2022] GET /identity/v3/projects/2635535fcc444c74b40bb9c46956c259 => generated 302 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:44.502455 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67db52a7-39c2-47f3-a240-36dac318ba0b None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:44.581037 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b528670-abac-4ec6-8fea-60eb5d5d7e6f None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:44.588442 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:44.606693 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67db52a7-39c2-47f3-a240-36dac318ba0b None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:44.616559 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d76dede-6e81-47b6-b2c4-b8d87b9ae1dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:44.617208 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d76dede-6e81-47b6-b2c4-b8d87b9ae1dc None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:44.617487 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d76dede-6e81-47b6-b2c4-b8d87b9ae1dc None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:44.617779 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d76dede-6e81-47b6-b2c4-b8d87b9ae1dc None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:44.618513 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d76dede-6e81-47b6-b2c4-b8d87b9ae1dc None admin] RBAC: Adding query filter params (group.id=205c648ef17a42ba92c3a3cbb750dc74, role.id=0487ef7e943e4701915cb8ce0f2899d7, scope.project.id=2635535fcc444c74b40bb9c46956c259) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:44.618817 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d76dede-6e81-47b6-b2c4-b8d87b9ae1dc None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:44.620283 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d76dede-6e81-47b6-b2c4-b8d87b9ae1dc None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:44.626332 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 219/448] 10.222.0.42 () {56 vars in 1376 bytes} [Sun Aug 28 00:41:44 2022] GET /identity/v3/role_assignments?group.id=205c648ef17a42ba92c3a3cbb750dc74&scope.project.id=2635535fcc444c74b40bb9c46956c259&role.id=0487ef7e943e4701915cb8ce0f2899d7 => generated 611 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:44.627361 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d76dede-6e81-47b6-b2c4-b8d87b9ae1dc None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:44.647077 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9b5f26a-9de8-4642-ac3b-5fe56c30911c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:44.647748 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9b5f26a-9de8-4642-ac3b-5fe56c30911c None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:44.647985 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9b5f26a-9de8-4642-ac3b-5fe56c30911c None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:44.648256 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9b5f26a-9de8-4642-ac3b-5fe56c30911c None admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:44.648994 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9b5f26a-9de8-4642-ac3b-5fe56c30911c None admin] RBAC: Adding query filter params (name=admin) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:44.649294 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9b5f26a-9de8-4642-ac3b-5fe56c30911c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:44.650719 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9b5f26a-9de8-4642-ac3b-5fe56c30911c None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:44.660837 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 230/449] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Aug 28 00:41:44 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:44.666440 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9b5f26a-9de8-4642-ac3b-5fe56c30911c None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:44.679158 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:44.746936 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d76dede-6e81-47b6-b2c4-b8d87b9ae1dc None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:44.782128 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9b5f26a-9de8-4642-ac3b-5fe56c30911c None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:44.786737 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97982abd-650b-4b93-9cff-4f258b0e6bf8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:44.786737 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97982abd-650b-4b93-9cff-4f258b0e6bf8 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:44.787004 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97982abd-650b-4b93-9cff-4f258b0e6bf8 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:44.787226 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97982abd-650b-4b93-9cff-4f258b0e6bf8 None admin] PATH_INFO: `/v3/groups/205c648ef17a42ba92c3a3cbb750dc74` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:44.788276 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97982abd-650b-4b93-9cff-4f258b0e6bf8 None admin] RBAC: Authorizing `identity:get_group(group_id=205c648ef17a42ba92c3a3cbb750dc74)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:44.789751 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97982abd-650b-4b93-9cff-4f258b0e6bf8 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:44.791341 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 220/450] 10.222.0.42 () {56 vars in 1180 bytes} [Sun Aug 28 00:41:44 2022] GET /identity/v3/groups/205c648ef17a42ba92c3a3cbb750dc74 => generated 226 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:44.797378 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97982abd-650b-4b93-9cff-4f258b0e6bf8 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:44.823286 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6665e36d-cbf8-4eac-aab9-792ddd164549 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:44.823904 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6665e36d-cbf8-4eac-aab9-792ddd164549 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:44.824184 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6665e36d-cbf8-4eac-aab9-792ddd164549 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:44.824453 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6665e36d-cbf8-4eac-aab9-792ddd164549 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:44.849080 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:44.917930 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6665e36d-cbf8-4eac-aab9-792ddd164549 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:44.919773 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97982abd-650b-4b93-9cff-4f258b0e6bf8 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:44.919900 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6665e36d-cbf8-4eac-aab9-792ddd164549 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:44.930882 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:44.958941 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3ebeac6-c9e8-4212-86de-4568ac17e4f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:44.959594 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3ebeac6-c9e8-4212-86de-4568ac17e4f4 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:44.960343 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3ebeac6-c9e8-4212-86de-4568ac17e4f4 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:44.960343 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3ebeac6-c9e8-4212-86de-4568ac17e4f4 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:44.960816 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3ebeac6-c9e8-4212-86de-4568ac17e4f4 None admin] RBAC: Adding query filter params (group.id=205c648ef17a42ba92c3a3cbb750dc74, role.id=05caf0846a15452b9a6a79fa59c7fa5a, scope.project.id=2635535fcc444c74b40bb9c46956c259) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:44.961152 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3ebeac6-c9e8-4212-86de-4568ac17e4f4 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:44.962888 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3ebeac6-c9e8-4212-86de-4568ac17e4f4 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:44.970831 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 221/451] 10.222.0.42 () {56 vars in 1376 bytes} [Sun Aug 28 00:41:44 2022] GET /identity/v3/role_assignments?group.id=205c648ef17a42ba92c3a3cbb750dc74&scope.project.id=2635535fcc444c74b40bb9c46956c259&role.id=05caf0846a15452b9a6a79fa59c7fa5a => generated 611 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 28 00:41:45.040318 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 231/452] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Aug 28 00:41:44 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 374 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Aug 28 00:41:45.046371 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3ebeac6-c9e8-4212-86de-4568ac17e4f4 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:45.158469 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3ebeac6-c9e8-4212-86de-4568ac17e4f4 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:45.160027 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:45.195279 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97ea58ad-aad0-4e6e-9e0d-cb9b136c4741 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:45.195848 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ea58ad-aad0-4e6e-9e0d-cb9b136c4741 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:45.196122 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ea58ad-aad0-4e6e-9e0d-cb9b136c4741 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:45.196389 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ea58ad-aad0-4e6e-9e0d-cb9b136c4741 None admin] PATH_INFO: `/v3/projects/cca4b1c3ff844aae827dd439e67820c7` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:45.197599 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97ea58ad-aad0-4e6e-9e0d-cb9b136c4741 None admin] RBAC: Authorizing `identity:get_project(project_id=cca4b1c3ff844aae827dd439e67820c7)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:45.199042 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97ea58ad-aad0-4e6e-9e0d-cb9b136c4741 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:45.200689 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 222/453] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Aug 28 00:41:45 2022] GET /identity/v3/projects/cca4b1c3ff844aae827dd439e67820c7 => generated 344 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:45.206954 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6665e36d-cbf8-4eac-aab9-792ddd164549 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:45.310172 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6665e36d-cbf8-4eac-aab9-792ddd164549 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:45.320483 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:45.346398 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27734ef4-c04e-4a4b-8597-31d523a6bcbe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:45.347025 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27734ef4-c04e-4a4b-8597-31d523a6bcbe None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:45.347314 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27734ef4-c04e-4a4b-8597-31d523a6bcbe None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:45.347578 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27734ef4-c04e-4a4b-8597-31d523a6bcbe None admin] PATH_INFO: `/v3/groups/189372dfaef04a538fb661ce5976dcc8` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:45.348675 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27734ef4-c04e-4a4b-8597-31d523a6bcbe None admin] RBAC: Authorizing `identity:get_group(group_id=189372dfaef04a538fb661ce5976dcc8)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:45.350173 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27734ef4-c04e-4a4b-8597-31d523a6bcbe None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:45.351773 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 232/454] 10.222.0.42 () {56 vars in 1180 bytes} [Sun Aug 28 00:41:45 2022] GET /identity/v3/groups/189372dfaef04a538fb661ce5976dcc8 => generated 229 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:45.357815 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97ea58ad-aad0-4e6e-9e0d-cb9b136c4741 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:45.467504 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97ea58ad-aad0-4e6e-9e0d-cb9b136c4741 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:45.508007 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-745390e6-06cd-4def-a3ba-4f6b31480cdf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:45.508608 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-745390e6-06cd-4def-a3ba-4f6b31480cdf None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:45.508883 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-745390e6-06cd-4def-a3ba-4f6b31480cdf None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:45.509189 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-745390e6-06cd-4def-a3ba-4f6b31480cdf None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:45.509907 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-745390e6-06cd-4def-a3ba-4f6b31480cdf None admin] RBAC: Adding query filter params (group.id=189372dfaef04a538fb661ce5976dcc8, role.id=f08d6043261747e3b4763c744eef7dfb, scope.project.id=cca4b1c3ff844aae827dd439e67820c7) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:45.510212 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-745390e6-06cd-4def-a3ba-4f6b31480cdf None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:45.511649 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-745390e6-06cd-4def-a3ba-4f6b31480cdf None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:45.517597 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 223/455] 10.222.0.42 () {56 vars in 1376 bytes} [Sun Aug 28 00:41:45 2022] GET /identity/v3/role_assignments?group.id=189372dfaef04a538fb661ce5976dcc8&scope.project.id=cca4b1c3ff844aae827dd439e67820c7&role.id=f08d6043261747e3b4763c744eef7dfb => generated 611 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:47.884803 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e0c09dc-d803-40c5-816a-dbde9f377628 None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:47.885871 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e0c09dc-d803-40c5-816a-dbde9f377628 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:47.886360 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e0c09dc-d803-40c5-816a-dbde9f377628 None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:47.888373 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 233/456] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:41:47 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:41:47.893702 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-754a9bd1-443f-45c7-98fa-b06f897704cd None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:47.894567 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-754a9bd1-443f-45c7-98fa-b06f897704cd None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:47.895056 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-754a9bd1-443f-45c7-98fa-b06f897704cd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:47.913885 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-754a9bd1-443f-45c7-98fa-b06f897704cd None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:41:47.915404 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-754a9bd1-443f-45c7-98fa-b06f897704cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:41:47.956071 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:48.006014 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 224/457] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:41:47 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 28 00:41:48.075566 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a35f0224-2d8a-4d57-a464-a7dbfc66293d None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:48.075900 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a35f0224-2d8a-4d57-a464-a7dbfc66293d None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:48.076164 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a35f0224-2d8a-4d57-a464-a7dbfc66293d None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:48.077531 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 234/458] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:41:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:41:48.086447 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-443ce6ad-be51-4b26-9b3b-aadc7c34ddf6 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:48.086998 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-443ce6ad-be51-4b26-9b3b-aadc7c34ddf6 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:48.087490 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-443ce6ad-be51-4b26-9b3b-aadc7c34ddf6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:48.109481 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-443ce6ad-be51-4b26-9b3b-aadc7c34ddf6 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:41:48.111034 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-443ce6ad-be51-4b26-9b3b-aadc7c34ddf6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:41:48.210494 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 225/459] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:41:48 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 28 00:41:48.268539 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f486f7-76e2-4524-a9da-a5ce78751d6b None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:48.269110 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f486f7-76e2-4524-a9da-a5ce78751d6b None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:48.269554 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f486f7-76e2-4524-a9da-a5ce78751d6b None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:48.271043 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 235/460] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:41:48 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:41:48.276496 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d40b467-c6e7-4d5e-9f37-b918e74b63bc None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:48.276787 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d40b467-c6e7-4d5e-9f37-b918e74b63bc None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:48.277090 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d40b467-c6e7-4d5e-9f37-b918e74b63bc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:48.297134 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-3d40b467-c6e7-4d5e-9f37-b918e74b63bc None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:41:48.298633 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-3d40b467-c6e7-4d5e-9f37-b918e74b63bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:41:48.323390 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01f486f7-76e2-4524-a9da-a5ce78751d6b None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:48.391354 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 226/461] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:41:48 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 28 00:41:48.448504 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01f486f7-76e2-4524-a9da-a5ce78751d6b None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:48.466206 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3fa69eb-1a5e-43f8-a749-cbae55522c71 None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:48.466494 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3fa69eb-1a5e-43f8-a749-cbae55522c71 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:48.466685 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3fa69eb-1a5e-43f8-a749-cbae55522c71 None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:48.469075 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 227/462] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:41:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:41:48.473991 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1fc4314-9543-4cdc-954e-62a3d0551410 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:48.474273 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1fc4314-9543-4cdc-954e-62a3d0551410 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:48.474544 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1fc4314-9543-4cdc-954e-62a3d0551410 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:48.484556 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24d26b95-b857-4f06-816e-d4f626aa4a17 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:48.485220 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d26b95-b857-4f06-816e-d4f626aa4a17 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:48.485499 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d26b95-b857-4f06-816e-d4f626aa4a17 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:48.485767 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d26b95-b857-4f06-816e-d4f626aa4a17 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:48.493348 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-d1fc4314-9543-4cdc-954e-62a3d0551410 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:41:48.494614 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-d1fc4314-9543-4cdc-954e-62a3d0551410 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:41:48.496740 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24d26b95-b857-4f06-816e-d4f626aa4a17 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:48.498350 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24d26b95-b857-4f06-816e-d4f626aa4a17 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:48.514097 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-24d26b95-b857-4f06-816e-d4f626aa4a17 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:41:48.514097 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:41:48.514097 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:41:48.514097 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:41:48.514097 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:41:48.514097 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:41:48.514097 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:41:48.514097 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:41:48.514097 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:41:48.514097 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:41:48.514097 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:41:48.514097 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:41:48.514097 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 28 00:41:48.514097 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 28 00:41:48.514097 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 28 00:41:48.514097 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 28 00:41:48.515187 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:41:48.515187 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:41:48.515187 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:41:48.515187 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:41:48.515187 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:41:48.515187 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:41:48.515187 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:41:48.515187 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:41:48.515187 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:41:48.515187 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:41:48.515187 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:41:48.515187 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:41:48.515187 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:41:48.515187 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:41:48.515187 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:41:48.515187 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:41:48.515187 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 28 00:41:48.516053 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 28 00:41:48.516053 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:41:48.516053 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:41:48.516053 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 236/463] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:41:48 2022] GET /identity/v3/domains/Default => generated 87 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 28 00:41:48.523431 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:48.525250 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24d26b95-b857-4f06-816e-d4f626aa4a17 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:48.575486 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:48.581076 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 228/464] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:41:48 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 28 00:41:48.673518 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24d26b95-b857-4f06-816e-d4f626aa4a17 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:48.708812 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfea5a12-49f0-4d7d-bbe5-eccedf8914e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:48.709453 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfea5a12-49f0-4d7d-bbe5-eccedf8914e4 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:48.709733 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfea5a12-49f0-4d7d-bbe5-eccedf8914e4 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:48.710055 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfea5a12-49f0-4d7d-bbe5-eccedf8914e4 None admin] PATH_INFO: `/v3/domains` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:48.710709 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfea5a12-49f0-4d7d-bbe5-eccedf8914e4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:48.711061 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfea5a12-49f0-4d7d-bbe5-eccedf8914e4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:48.712128 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1fc4314-9543-4cdc-954e-62a3d0551410 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:48.712443 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfea5a12-49f0-4d7d-bbe5-eccedf8914e4 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:48.725736 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 237/465] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Aug 28 00:41:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:48.726633 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfea5a12-49f0-4d7d-bbe5-eccedf8914e4 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:48.824463 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1fc4314-9543-4cdc-954e-62a3d0551410 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:48.834111 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:48.847104 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfea5a12-49f0-4d7d-bbe5-eccedf8914e4 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:48.866272 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91bdb184-897c-4633-8ae3-e06c527e5c0a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:48.866910 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91bdb184-897c-4633-8ae3-e06c527e5c0a None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:48.867189 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91bdb184-897c-4633-8ae3-e06c527e5c0a None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:48.867734 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91bdb184-897c-4633-8ae3-e06c527e5c0a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:48.877985 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91bdb184-897c-4633-8ae3-e06c527e5c0a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:48.879369 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91bdb184-897c-4633-8ae3-e06c527e5c0a None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:48.882769 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c11f4c96-8b52-4bac-8897-0ca856f5e075 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:48.883398 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c11f4c96-8b52-4bac-8897-0ca856f5e075 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:48.883675 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c11f4c96-8b52-4bac-8897-0ca856f5e075 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:48.883945 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c11f4c96-8b52-4bac-8897-0ca856f5e075 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:48.894558 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c11f4c96-8b52-4bac-8897-0ca856f5e075 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:48.895235 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-91bdb184-897c-4633-8ae3-e06c527e5c0a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:41:48.895235 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:41:48.895235 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:41:48.895235 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:41:48.895235 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:41:48.895235 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:41:48.895235 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:41:48.895235 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:41:48.895235 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:41:48.895235 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:41:48.895235 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:41:48.895235 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:41:48.895235 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 28 00:41:48.895235 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 28 00:41:48.895235 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 28 00:41:48.895235 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 28 00:41:48.896342 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:41:48.896342 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:41:48.896342 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:41:48.896342 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:41:48.896342 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:41:48.896342 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:41:48.896342 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:41:48.896342 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:41:48.896342 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:41:48.896342 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:41:48.896342 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:41:48.896342 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:41:48.896342 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:41:48.896342 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:41:48.896342 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:41:48.896342 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:41:48.896342 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 28 00:41:48.897215 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 28 00:41:48.897215 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:41:48.897215 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:41:48.897375 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c11f4c96-8b52-4bac-8897-0ca856f5e075 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:48.897443 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 229/466] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:41:48 2022] GET /identity/v3/domains/Default => generated 87 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:41:48.897443 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91bdb184-897c-4633-8ae3-e06c527e5c0a None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:48.906865 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:48.917121 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-c11f4c96-8b52-4bac-8897-0ca856f5e075 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:41:48.917121 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:41:48.917121 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:41:48.917121 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:41:48.917121 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:41:48.917121 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:41:48.917121 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:41:48.917121 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:41:48.917121 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:41:48.917121 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:41:48.917121 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:41:48.917121 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:41:48.917121 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 28 00:41:48.917121 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 28 00:41:48.917121 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 28 00:41:48.917121 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 28 00:41:48.918322 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:41:48.918322 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:41:48.918322 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:41:48.918322 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:41:48.918322 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:41:48.918322 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:41:48.918322 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:41:48.918322 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:41:48.918322 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:41:48.918322 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:41:48.918322 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:41:48.918322 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:41:48.918322 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:41:48.918322 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:41:48.918322 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:41:48.918322 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:41:48.918322 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 28 00:41:48.919402 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 28 00:41:48.919402 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:41:48.919402 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:41:48.921131 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 238/467] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:41:48 2022] GET /identity/v3/domains/Default => generated 87 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:41:48.922117 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c11f4c96-8b52-4bac-8897-0ca856f5e075 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:49.009984 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91bdb184-897c-4633-8ae3-e06c527e5c0a None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:49.030647 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c11f4c96-8b52-4bac-8897-0ca856f5e075 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:49.047046 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63b09fc2-5112-4e51-9475-ad8f87866267 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:49.048025 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63b09fc2-5112-4e51-9475-ad8f87866267 None admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:49.048498 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63b09fc2-5112-4e51-9475-ad8f87866267 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:49.048918 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63b09fc2-5112-4e51-9475-ad8f87866267 None admin] PATH_INFO: `/v3/users` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:49.049618 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63b09fc2-5112-4e51-9475-ad8f87866267 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:49.051078 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63b09fc2-5112-4e51-9475-ad8f87866267 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:49.067403 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60bf5ccd-c68a-4569-8819-88be3520fbdc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:49.068185 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60bf5ccd-c68a-4569-8819-88be3520fbdc None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:49.068469 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60bf5ccd-c68a-4569-8819-88be3520fbdc None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:49.068743 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60bf5ccd-c68a-4569-8819-88be3520fbdc None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:49.077286 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 230/468] 10.222.0.42 () {60 vars in 1132 bytes} [Sun Aug 28 00:41:48 2022] POST /identity/v3/users => generated 248 bytes in 175 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:41:49.077286 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63b09fc2-5112-4e51-9475-ad8f87866267 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:49.087810 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:49.092664 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60bf5ccd-c68a-4569-8819-88be3520fbdc None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:49.095860 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60bf5ccd-c68a-4569-8819-88be3520fbdc None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:49.137835 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-60bf5ccd-c68a-4569-8819-88be3520fbdc None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:41:49.137835 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:41:49.137835 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:41:49.137835 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:41:49.137835 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:41:49.137835 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:41:49.137835 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:41:49.137835 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:41:49.137835 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:41:49.137835 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:41:49.137835 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:41:49.137835 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:41:49.137835 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 28 00:41:49.137835 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 28 00:41:49.137835 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 28 00:41:49.137835 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 28 00:41:49.138885 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:41:49.138885 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:41:49.138885 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:41:49.138885 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:41:49.138885 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:41:49.138885 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:41:49.138885 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:41:49.138885 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:41:49.138885 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:41:49.138885 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:41:49.138885 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:41:49.138885 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:41:49.138885 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:41:49.138885 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:41:49.138885 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:41:49.138885 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:41:49.138885 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 28 00:41:49.139747 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 28 00:41:49.139747 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:41:49.139747 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:41:49.142435 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 239/469] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:41:48 2022] GET /identity/v3/domains/Default => generated 87 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:41:49.143619 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60bf5ccd-c68a-4569-8819-88be3520fbdc None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:49.169375 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:49.222743 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63b09fc2-5112-4e51-9475-ad8f87866267 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:49.268259 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee0dec04-10f3-4848-a90a-3ec9d020c3eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:49.268569 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee0dec04-10f3-4848-a90a-3ec9d020c3eb None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:49.269197 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee0dec04-10f3-4848-a90a-3ec9d020c3eb None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:49.269197 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee0dec04-10f3-4848-a90a-3ec9d020c3eb None admin] PATH_INFO: `/v3/domains` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:49.270357 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee0dec04-10f3-4848-a90a-3ec9d020c3eb None admin] RBAC: Adding query filter params (name=Default) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:49.270357 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee0dec04-10f3-4848-a90a-3ec9d020c3eb None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:49.271487 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee0dec04-10f3-4848-a90a-3ec9d020c3eb None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:49.286774 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 231/470] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Aug 28 00:41:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:49.287973 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee0dec04-10f3-4848-a90a-3ec9d020c3eb None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:49.302136 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60bf5ccd-c68a-4569-8819-88be3520fbdc None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:49.348555 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-688177d8-d096-46e9-be6f-1b2cc10c2796 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:49.350657 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-688177d8-d096-46e9-be6f-1b2cc10c2796 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:49.350657 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-688177d8-d096-46e9-be6f-1b2cc10c2796 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:49.350657 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-688177d8-d096-46e9-be6f-1b2cc10c2796 None admin] PATH_INFO: `/v3/domains` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:49.351106 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-688177d8-d096-46e9-be6f-1b2cc10c2796 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:49.351573 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-688177d8-d096-46e9-be6f-1b2cc10c2796 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:49.354700 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-688177d8-d096-46e9-be6f-1b2cc10c2796 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:49.370559 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 240/471] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Aug 28 00:41:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:49.372017 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-688177d8-d096-46e9-be6f-1b2cc10c2796 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:49.430628 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee0dec04-10f3-4848-a90a-3ec9d020c3eb None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:49.440792 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:49.460665 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:49.481009 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2876b8ab-ea00-4869-b4c7-2ee2d5a8cebc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:49.481746 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2876b8ab-ea00-4869-b4c7-2ee2d5a8cebc None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:49.482063 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2876b8ab-ea00-4869-b4c7-2ee2d5a8cebc None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:49.482344 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2876b8ab-ea00-4869-b4c7-2ee2d5a8cebc None admin] PATH_INFO: `/v3/domains` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:49.483001 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2876b8ab-ea00-4869-b4c7-2ee2d5a8cebc None admin] RBAC: Adding query filter params (name=Default) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:49.483303 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2876b8ab-ea00-4869-b4c7-2ee2d5a8cebc None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:49.484846 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2876b8ab-ea00-4869-b4c7-2ee2d5a8cebc None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:49.497683 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 232/472] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Aug 28 00:41:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:49.498529 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2876b8ab-ea00-4869-b4c7-2ee2d5a8cebc None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:49.509645 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-688177d8-d096-46e9-be6f-1b2cc10c2796 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:49.555427 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fda24ac2-6435-4bbf-bc18-f1a08b354cf4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:49.556099 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda24ac2-6435-4bbf-bc18-f1a08b354cf4 None admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:49.556513 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda24ac2-6435-4bbf-bc18-f1a08b354cf4 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:49.556785 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda24ac2-6435-4bbf-bc18-f1a08b354cf4 None admin] PATH_INFO: `/v3/users` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:49.557726 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fda24ac2-6435-4bbf-bc18-f1a08b354cf4 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:49.559633 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fda24ac2-6435-4bbf-bc18-f1a08b354cf4 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:49.583419 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 241/473] 10.222.0.42 () {60 vars in 1132 bytes} [Sun Aug 28 00:41:49 2022] POST /identity/v3/users => generated 246 bytes in 212 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:41:49.584482 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fda24ac2-6435-4bbf-bc18-f1a08b354cf4 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:49.622428 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2876b8ab-ea00-4869-b4c7-2ee2d5a8cebc None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:49.666602 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a686cd79-51b2-4170-a071-ff6a9d1b3e42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:49.667380 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a686cd79-51b2-4170-a071-ff6a9d1b3e42 None admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:49.667755 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a686cd79-51b2-4170-a071-ff6a9d1b3e42 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:49.668124 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a686cd79-51b2-4170-a071-ff6a9d1b3e42 None admin] PATH_INFO: `/v3/users` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:49.669087 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a686cd79-51b2-4170-a071-ff6a9d1b3e42 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:49.670907 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a686cd79-51b2-4170-a071-ff6a9d1b3e42 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:49.706488 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 233/474] 10.222.0.42 () {60 vars in 1132 bytes} [Sun Aug 28 00:41:49 2022] POST /identity/v3/users => generated 248 bytes in 208 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:41:49.707950 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c104b5a7-92cd-4fc9-80bc-437ab49c8e41 None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:49.708240 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c104b5a7-92cd-4fc9-80bc-437ab49c8e41 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:49.708510 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c104b5a7-92cd-4fc9-80bc-437ab49c8e41 None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:49.709917 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 234/475] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:41:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:41:49.716111 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba7e9f0c-e6c9-4ad9-8aa7-c179303c0b6e None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:49.716395 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba7e9f0c-e6c9-4ad9-8aa7-c179303c0b6e None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:49.716668 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba7e9f0c-e6c9-4ad9-8aa7-c179303c0b6e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:49.735820 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fda24ac2-6435-4bbf-bc18-f1a08b354cf4 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:49.739837 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-ba7e9f0c-e6c9-4ad9-8aa7-c179303c0b6e None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:41:49.741128 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-ba7e9f0c-e6c9-4ad9-8aa7-c179303c0b6e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:41:49.747690 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:49.780152 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbe5093f-97c0-4a19-8a88-7308f934b0a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:49.781032 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe5093f-97c0-4a19-8a88-7308f934b0a7 None admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:49.781419 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe5093f-97c0-4a19-8a88-7308f934b0a7 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:49.781791 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe5093f-97c0-4a19-8a88-7308f934b0a7 None admin] PATH_INFO: `/v3/users` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:49.782269 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:49.782463 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbe5093f-97c0-4a19-8a88-7308f934b0a7 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:49.783967 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbe5093f-97c0-4a19-8a88-7308f934b0a7 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:49.822990 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 242/476] 10.222.0.42 () {60 vars in 1132 bytes} [Sun Aug 28 00:41:49 2022] POST /identity/v3/users => generated 249 bytes in 239 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:41:49.853934 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 235/477] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:41:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 28 00:41:50.139935 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbe5093f-97c0-4a19-8a88-7308f934b0a7 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:50.252170 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbe5093f-97c0-4a19-8a88-7308f934b0a7 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:50.289280 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d312ef2d-ca59-4fdb-9f45-a6f6c0645949 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:50.290266 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d312ef2d-ca59-4fdb-9f45-a6f6c0645949 None admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:50.290793 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d312ef2d-ca59-4fdb-9f45-a6f6c0645949 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:50.291254 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d312ef2d-ca59-4fdb-9f45-a6f6c0645949 None admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:50.292126 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d312ef2d-ca59-4fdb-9f45-a6f6c0645949 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:50.293744 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d312ef2d-ca59-4fdb-9f45-a6f6c0645949 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:50.302086 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.sql.core [None req-d312ef2d-ca59-4fdb-9f45-a6f6c0645949 None admin] Conflict role: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Aug 28 00:41:50.302086 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Aug 28 00:41:50.302086 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [parameters: {'id': '2312c8ffe1704b638f4e2ae24ec61588', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Aug 28 00:41:50.302086 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81507) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Aug 28 00:41:50.321209 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-d312ef2d-ca59-4fdb-9f45-a6f6c0645949 None admin] Conflict occurred attempting to store role - Duplicate entry found with name anotherrole.: keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Aug 28 00:41:50.321209 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:41:50.321209 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Aug 28 00:41:50.321209 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application self.dialect.do_execute( Aug 28 00:41:50.321209 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Aug 28 00:41:50.321209 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Aug 28 00:41:50.321209 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Aug 28 00:41:50.321209 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application result = self._query(query) Aug 28 00:41:50.321209 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Aug 28 00:41:50.321209 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application conn.query(q) Aug 28 00:41:50.321209 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Aug 28 00:41:50.321209 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Aug 28 00:41:50.321209 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Aug 28 00:41:50.321209 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application result.read() Aug 28 00:41:50.321209 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Aug 28 00:41:50.322145 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Aug 28 00:41:50.322145 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Aug 28 00:41:50.322145 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application packet.raise_for_error() Aug 28 00:41:50.322145 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Aug 28 00:41:50.322145 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Aug 28 00:41:50.322145 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Aug 28 00:41:50.322145 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Aug 28 00:41:50.322145 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Aug 28 00:41:50.322145 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:41:50.322145 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Aug 28 00:41:50.322145 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:41:50.322145 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:41:50.322145 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Aug 28 00:41:50.322145 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return method(*args, **kwargs) Aug 28 00:41:50.322145 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 33, in create_role Aug 28 00:41:50.322145 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return ref.to_dict() Aug 28 00:41:50.322145 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Aug 28 00:41:50.322145 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application next(self.gen) Aug 28 00:41:50.323026 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Aug 28 00:41:50.323026 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application yield resource Aug 28 00:41:50.323026 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Aug 28 00:41:50.323026 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application next(self.gen) Aug 28 00:41:50.323026 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Aug 28 00:41:50.323026 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application self.session.rollback() Aug 28 00:41:50.323026 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Aug 28 00:41:50.323026 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application self.force_reraise() Aug 28 00:41:50.323026 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Aug 28 00:41:50.323026 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise self.value Aug 28 00:41:50.323026 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Aug 28 00:41:50.323026 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Aug 28 00:41:50.323026 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Aug 28 00:41:50.323026 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application session.commit() Aug 28 00:41:50.323026 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Aug 28 00:41:50.323026 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Aug 28 00:41:50.323026 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Aug 28 00:41:50.324007 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application self._prepare_impl() Aug 28 00:41:50.324007 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Aug 28 00:41:50.324007 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application self.session.flush() Aug 28 00:41:50.324007 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Aug 28 00:41:50.324007 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application self._flush(objects) Aug 28 00:41:50.324007 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Aug 28 00:41:50.324007 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Aug 28 00:41:50.324007 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Aug 28 00:41:50.324007 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application compat.raise_( Aug 28 00:41:50.324007 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Aug 28 00:41:50.324007 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception Aug 28 00:41:50.324007 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Aug 28 00:41:50.324007 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application flush_context.execute() Aug 28 00:41:50.324007 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Aug 28 00:41:50.324007 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rec.execute(self) Aug 28 00:41:50.324007 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Aug 28 00:41:50.324007 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Aug 28 00:41:50.324875 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Aug 28 00:41:50.324875 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application _emit_insert_statements( Aug 28 00:41:50.324875 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Aug 28 00:41:50.324875 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application c = connection._execute_20( Aug 28 00:41:50.324875 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Aug 28 00:41:50.324875 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Aug 28 00:41:50.324875 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Aug 28 00:41:50.324875 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Aug 28 00:41:50.324875 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Aug 28 00:41:50.324875 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ret = self._execute_context( Aug 28 00:41:50.324875 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Aug 28 00:41:50.324875 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Aug 28 00:41:50.324875 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Aug 28 00:41:50.324875 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Aug 28 00:41:50.324875 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Aug 28 00:41:50.324875 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception Aug 28 00:41:50.327371 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Aug 28 00:41:50.327371 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application self.dialect.do_execute( Aug 28 00:41:50.327371 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Aug 28 00:41:50.327371 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Aug 28 00:41:50.327371 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Aug 28 00:41:50.327371 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application result = self._query(query) Aug 28 00:41:50.327371 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Aug 28 00:41:50.327371 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application conn.query(q) Aug 28 00:41:50.327371 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Aug 28 00:41:50.327371 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Aug 28 00:41:50.327371 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Aug 28 00:41:50.327371 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application result.read() Aug 28 00:41:50.327371 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Aug 28 00:41:50.327371 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Aug 28 00:41:50.327371 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Aug 28 00:41:50.327371 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application packet.raise_for_error() Aug 28 00:41:50.329655 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Aug 28 00:41:50.329655 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Aug 28 00:41:50.329655 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Aug 28 00:41:50.329655 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Aug 28 00:41:50.329655 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Aug 28 00:41:50.329655 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Aug 28 00:41:50.329655 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [parameters: {'id': '2312c8ffe1704b638f4e2ae24ec61588', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Aug 28 00:41:50.329655 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Aug 28 00:41:50.329655 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:41:50.329655 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 28 00:41:50.329655 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:41:50.329655 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:41:50.329655 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:41:50.329655 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:41:50.329655 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:41:50.329655 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:41:50.332894 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:41:50.332894 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:41:50.332894 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:41:50.332894 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:41:50.332894 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:41:50.332894 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:41:50.332894 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 115, in post Aug 28 00:41:50.332894 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ref = PROVIDERS.role_api.create_role( Aug 28 00:41:50.332894 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:41:50.332894 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:41:50.332894 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1298, in create_role Aug 28 00:41:50.332894 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ret = self.driver.create_role(role_id, role) Aug 28 00:41:50.332894 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Aug 28 00:41:50.332894 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Aug 28 00:41:50.332894 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Aug 28 00:41:50.332894 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:41:50.336817 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 243/478] 10.222.0.42 () {60 vars in 1131 bytes} [Sun Aug 28 00:41:50 2022] POST /identity/v3/roles => generated 144 bytes in 196 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:41:50.343787 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba7e9f0c-e6c9-4ad9-8aa7-c179303c0b6e None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:50.508250 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba7e9f0c-e6c9-4ad9-8aa7-c179303c0b6e None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:50.570697 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f6b564f-9fed-4aee-85d1-5a04d8ffa65d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:50.571362 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f6b564f-9fed-4aee-85d1-5a04d8ffa65d None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:50.571637 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f6b564f-9fed-4aee-85d1-5a04d8ffa65d None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:50.571907 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f6b564f-9fed-4aee-85d1-5a04d8ffa65d None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:50.593098 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f6b564f-9fed-4aee-85d1-5a04d8ffa65d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 28 00:41:50.598191 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-2f6b564f-9fed-4aee-85d1-5a04d8ffa65d None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 28 00:41:50.598191 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:41:50.598191 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:41:50.598191 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:41:50.598191 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:41:50.598191 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:41:50.598191 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:41:50.598191 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:41:50.598191 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:41:50.598191 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:41:50.598191 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:41:50.598191 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:41:50.598191 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 28 00:41:50.598191 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 28 00:41:50.598191 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 28 00:41:50.598191 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 28 00:41:50.599063 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 28 00:41:50.599063 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 28 00:41:50.599063 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 28 00:41:50.599063 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 28 00:41:50.599063 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:41:50.599063 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:41:50.599063 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:41:50.599063 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:41:50.599063 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:41:50.599063 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:41:50.599063 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:41:50.599063 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:41:50.599063 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:41:50.599063 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:41:50.599063 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:41:50.599063 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:41:50.599845 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:41:50.599845 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:41:50.599845 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:41:50.599845 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:41:50.599845 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 28 00:41:50.599845 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 28 00:41:50.599845 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 28 00:41:50.599845 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 28 00:41:50.599845 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 28 00:41:50.599845 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 28 00:41:50.599845 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Aug 28 00:41:50.599845 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:41:50.600434 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 236/479] 10.222.0.42 () {56 vars in 1114 bytes} [Sun Aug 28 00:41:50 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 257 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:41:50.605047 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d312ef2d-ca59-4fdb-9f45-a6f6c0645949 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:50.659860 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:50.758932 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d312ef2d-ca59-4fdb-9f45-a6f6c0645949 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:50.802438 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39d56d5b-ecfb-4ecf-a760-0da0151a5ff5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:50.803122 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39d56d5b-ecfb-4ecf-a760-0da0151a5ff5 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:50.803404 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39d56d5b-ecfb-4ecf-a760-0da0151a5ff5 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:50.803673 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39d56d5b-ecfb-4ecf-a760-0da0151a5ff5 None admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:50.804353 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39d56d5b-ecfb-4ecf-a760-0da0151a5ff5 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:50.804652 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39d56d5b-ecfb-4ecf-a760-0da0151a5ff5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:50.806067 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39d56d5b-ecfb-4ecf-a760-0da0151a5ff5 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:50.819778 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 244/480] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Aug 28 00:41:50 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:51.442930 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a78d65ee-0bd6-4303-958c-47e2fc79c645 None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:51.443481 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a78d65ee-0bd6-4303-958c-47e2fc79c645 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:51.443481 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a78d65ee-0bd6-4303-958c-47e2fc79c645 None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:51.445036 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 237/481] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:41:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:41:51.450619 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78dd9a4d-8a02-45bd-8495-dd8d3f6b122c None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:51.451065 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78dd9a4d-8a02-45bd-8495-dd8d3f6b122c None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:51.451166 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78dd9a4d-8a02-45bd-8495-dd8d3f6b122c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:51.480283 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-78dd9a4d-8a02-45bd-8495-dd8d3f6b122c None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:41:51.481736 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-78dd9a4d-8a02-45bd-8495-dd8d3f6b122c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:41:51.567122 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 245/482] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:41:51 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 28 00:41:51.916142 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a78d65ee-0bd6-4303-958c-47e2fc79c645 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:51.951361 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:52.047937 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a78d65ee-0bd6-4303-958c-47e2fc79c645 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:52.085499 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2660cfe0-5849-41c7-ac90-ca64cb3a61c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:52.086123 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2660cfe0-5849-41c7-ac90-ca64cb3a61c5 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:52.086397 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2660cfe0-5849-41c7-ac90-ca64cb3a61c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:52.086662 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2660cfe0-5849-41c7-ac90-ca64cb3a61c5 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:52.095728 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24452d1a-2572-46dd-a9fe-7a2bf59a8f0c None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:52.096048 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24452d1a-2572-46dd-a9fe-7a2bf59a8f0c None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:52.096272 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24452d1a-2572-46dd-a9fe-7a2bf59a8f0c None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:52.097904 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 246/483] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:41:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:41:52.098553 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2660cfe0-5849-41c7-ac90-ca64cb3a61c5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 28 00:41:52.103412 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d606ec4-2edc-49c2-be5e-c4a98e921009 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:52.103677 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-2660cfe0-5849-41c7-ac90-ca64cb3a61c5 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 28 00:41:52.103677 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:41:52.103677 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:41:52.103677 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:41:52.103677 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:41:52.103677 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:41:52.103677 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:41:52.103677 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:41:52.103677 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:41:52.103677 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:41:52.103677 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:41:52.103677 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:41:52.103677 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 28 00:41:52.103677 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 28 00:41:52.103677 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 28 00:41:52.103677 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 28 00:41:52.104544 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 28 00:41:52.104544 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 28 00:41:52.104544 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 28 00:41:52.104544 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 28 00:41:52.104544 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:41:52.104544 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:41:52.104544 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:41:52.104544 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:41:52.104544 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:41:52.104544 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:41:52.104544 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:41:52.104544 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:41:52.104544 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:41:52.104544 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:41:52.104544 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:41:52.104544 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:41:52.105917 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:41:52.105917 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:41:52.105917 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:41:52.105917 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:41:52.105917 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 28 00:41:52.105917 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 28 00:41:52.105917 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 28 00:41:52.105917 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 28 00:41:52.105917 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 28 00:41:52.105917 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 28 00:41:52.105917 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 28 00:41:52.105917 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:41:52.106845 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d606ec4-2edc-49c2-be5e-c4a98e921009 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:52.106845 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d606ec4-2edc-49c2-be5e-c4a98e921009 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:52.107004 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 238/484] 10.222.0.42 () {56 vars in 1102 bytes} [Sun Aug 28 00:41:51 2022] GET /identity/v3/roles/service => generated 85 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:41:52.115458 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2660cfe0-5849-41c7-ac90-ca64cb3a61c5 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:52.146394 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-9d606ec4-2edc-49c2-be5e-c4a98e921009 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:41:52.147601 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-9d606ec4-2edc-49c2-be5e-c4a98e921009 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:41:52.171412 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:52.245225 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 247/485] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:41:52 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 142 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 28 00:41:52.246711 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8e50b71-692e-459a-a51d-8ba3cac56852 None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:52.247070 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8e50b71-692e-459a-a51d-8ba3cac56852 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:52.247414 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8e50b71-692e-459a-a51d-8ba3cac56852 None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:52.247884 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:52.248996 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 248/486] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:41:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:41:52.256143 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f19e30-5078-424b-8691-26e74e545b30 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:52.256424 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f19e30-5078-424b-8691-26e74e545b30 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:52.256694 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f19e30-5078-424b-8691-26e74e545b30 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:52.283657 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2660cfe0-5849-41c7-ac90-ca64cb3a61c5 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:52.297769 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-53f19e30-5078-424b-8691-26e74e545b30 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:41:52.298989 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-53f19e30-5078-424b-8691-26e74e545b30 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:41:52.354518 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2a37618-abc8-4ea2-8469-83d9779efdf1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:52.355136 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2a37618-abc8-4ea2-8469-83d9779efdf1 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:52.355417 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2a37618-abc8-4ea2-8469-83d9779efdf1 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:52.355689 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2a37618-abc8-4ea2-8469-83d9779efdf1 None admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:52.356363 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2a37618-abc8-4ea2-8469-83d9779efdf1 None admin] RBAC: Adding query filter params (name=service) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:52.356662 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2a37618-abc8-4ea2-8469-83d9779efdf1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:52.358083 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2a37618-abc8-4ea2-8469-83d9779efdf1 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:52.378692 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 239/487] 10.222.0.42 () {56 vars in 1103 bytes} [Sun Aug 28 00:41:52 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 264 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:52.380333 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88f6fa9d-060b-4d4c-8b92-6ad2b15738a0 None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:52.380820 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88f6fa9d-060b-4d4c-8b92-6ad2b15738a0 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:52.381310 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88f6fa9d-060b-4d4c-8b92-6ad2b15738a0 None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:52.382790 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 240/488] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:41:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:41:52.387726 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88f6fa9d-060b-4d4c-8b92-6ad2b15738a0 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:52.420997 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 249/489] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:41:52 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 28 00:41:52.420997 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb3c7931-e458-4e96-a654-5570e3ffef82 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:52.420997 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb3c7931-e458-4e96-a654-5570e3ffef82 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:52.420997 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb3c7931-e458-4e96-a654-5570e3ffef82 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:52.441085 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-bb3c7931-e458-4e96-a654-5570e3ffef82 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:41:52.442426 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-bb3c7931-e458-4e96-a654-5570e3ffef82 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:41:52.524564 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 250/490] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:41:52 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 28 00:41:52.525879 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb3c7931-e458-4e96-a654-5570e3ffef82 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:52.541001 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:52.554519 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88f6fa9d-060b-4d4c-8b92-6ad2b15738a0 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:52.646350 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c94c313-50dc-4d8a-a89f-7f2dc0f7e83f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:52.647006 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c94c313-50dc-4d8a-a89f-7f2dc0f7e83f None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:52.647281 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c94c313-50dc-4d8a-a89f-7f2dc0f7e83f None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:52.647552 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c94c313-50dc-4d8a-a89f-7f2dc0f7e83f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:52.663766 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb3c7931-e458-4e96-a654-5570e3ffef82 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:52.673747 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c94c313-50dc-4d8a-a89f-7f2dc0f7e83f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:52.675330 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c94c313-50dc-4d8a-a89f-7f2dc0f7e83f None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:52.702291 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-6c94c313-50dc-4d8a-a89f-7f2dc0f7e83f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:41:52.702291 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:41:52.702291 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:41:52.702291 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:41:52.702291 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:41:52.702291 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:41:52.702291 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:41:52.702291 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:41:52.702291 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:41:52.702291 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:41:52.702291 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:41:52.702291 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:41:52.702291 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 28 00:41:52.702291 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 28 00:41:52.702291 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 28 00:41:52.702291 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 28 00:41:52.703270 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:41:52.703270 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:41:52.703270 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:41:52.703270 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:41:52.703270 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:41:52.703270 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:41:52.703270 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:41:52.703270 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:41:52.703270 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:41:52.703270 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:41:52.703270 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:41:52.703270 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:41:52.703270 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:41:52.703270 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:41:52.703270 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:41:52.703270 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:41:52.703270 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 28 00:41:52.704114 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 28 00:41:52.704114 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:41:52.704114 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:41:52.704275 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9175967c-0177-4683-a2f3-b5cd2919a95b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:52.704275 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9175967c-0177-4683-a2f3-b5cd2919a95b None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:52.704275 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9175967c-0177-4683-a2f3-b5cd2919a95b None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:52.704479 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 241/491] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:41:52 2022] GET /identity/v3/domains/Default => generated 87 bytes in 320 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:41:52.704544 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9175967c-0177-4683-a2f3-b5cd2919a95b None admin] PATH_INFO: `/v3/roles/service` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:52.709543 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c94c313-50dc-4d8a-a89f-7f2dc0f7e83f None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:52.715967 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:52.719395 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9175967c-0177-4683-a2f3-b5cd2919a95b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 28 00:41:52.726286 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-9175967c-0177-4683-a2f3-b5cd2919a95b None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 28 00:41:52.726286 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:41:52.726286 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:41:52.726286 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:41:52.726286 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:41:52.726286 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:41:52.726286 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:41:52.726286 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:41:52.726286 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:41:52.726286 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:41:52.726286 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:41:52.726286 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:41:52.726286 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 28 00:41:52.726286 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 28 00:41:52.726286 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 28 00:41:52.726286 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 28 00:41:52.728673 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 28 00:41:52.728673 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 28 00:41:52.728673 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 28 00:41:52.728673 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 28 00:41:52.728673 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:41:52.728673 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:41:52.728673 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:41:52.728673 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:41:52.728673 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:41:52.728673 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:41:52.728673 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:41:52.728673 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:41:52.728673 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:41:52.728673 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:41:52.728673 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:41:52.728673 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:41:52.730628 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:41:52.730628 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:41:52.730628 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:41:52.730628 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:41:52.730628 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 28 00:41:52.730628 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 28 00:41:52.730628 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 28 00:41:52.730628 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 28 00:41:52.730628 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 28 00:41:52.730628 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 28 00:41:52.730628 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 28 00:41:52.730628 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:41:52.730628 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 251/492] 10.222.0.42 () {56 vars in 1102 bytes} [Sun Aug 28 00:41:52 2022] GET /identity/v3/roles/service => generated 85 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:41:52.742907 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9175967c-0177-4683-a2f3-b5cd2919a95b None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:52.832611 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c94c313-50dc-4d8a-a89f-7f2dc0f7e83f None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:52.847832 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:52.876057 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b0d74c7-05dd-4483-bbe6-a7237ac7f91b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:52.876672 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b0d74c7-05dd-4483-bbe6-a7237ac7f91b None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:52.876949 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b0d74c7-05dd-4483-bbe6-a7237ac7f91b None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:52.877246 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b0d74c7-05dd-4483-bbe6-a7237ac7f91b None admin] PATH_INFO: `/v3/domains` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:52.877883 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b0d74c7-05dd-4483-bbe6-a7237ac7f91b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:52.878181 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b0d74c7-05dd-4483-bbe6-a7237ac7f91b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:52.879688 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b0d74c7-05dd-4483-bbe6-a7237ac7f91b None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:52.893591 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 242/493] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Aug 28 00:41:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:52.894259 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9175967c-0177-4683-a2f3-b5cd2919a95b None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:52.894526 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b0d74c7-05dd-4483-bbe6-a7237ac7f91b None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:52.943153 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ea7d4b5-af8b-4f1f-9f5f-cd85d66ef78d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:52.944468 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea7d4b5-af8b-4f1f-9f5f-cd85d66ef78d None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:52.944953 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea7d4b5-af8b-4f1f-9f5f-cd85d66ef78d None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:52.945695 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea7d4b5-af8b-4f1f-9f5f-cd85d66ef78d None admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:52.947339 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ea7d4b5-af8b-4f1f-9f5f-cd85d66ef78d None admin] RBAC: Adding query filter params (name=service) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:52.947651 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ea7d4b5-af8b-4f1f-9f5f-cd85d66ef78d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:52.949091 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ea7d4b5-af8b-4f1f-9f5f-cd85d66ef78d None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:52.958435 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 252/494] 10.222.0.42 () {56 vars in 1103 bytes} [Sun Aug 28 00:41:52 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:52.959049 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ea7d4b5-af8b-4f1f-9f5f-cd85d66ef78d None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:52.969756 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:53.017500 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b0d74c7-05dd-4483-bbe6-a7237ac7f91b None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:53.055595 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dd97fc1-148b-4cd4-99b1-6dcaaf7bbcc2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:53.056536 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd97fc1-148b-4cd4-99b1-6dcaaf7bbcc2 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:53.056810 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd97fc1-148b-4cd4-99b1-6dcaaf7bbcc2 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:53.057376 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd97fc1-148b-4cd4-99b1-6dcaaf7bbcc2 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:53.071763 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dd97fc1-148b-4cd4-99b1-6dcaaf7bbcc2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 28 00:41:53.078225 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-2dd97fc1-148b-4cd4-99b1-6dcaaf7bbcc2 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 28 00:41:53.078225 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:41:53.078225 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:41:53.078225 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:41:53.078225 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:41:53.078225 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:41:53.078225 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:41:53.078225 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:41:53.078225 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:41:53.078225 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:41:53.078225 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:41:53.078225 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:41:53.078225 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 28 00:41:53.078225 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 28 00:41:53.078225 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 28 00:41:53.078225 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 28 00:41:53.079130 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 28 00:41:53.079130 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 28 00:41:53.079130 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 28 00:41:53.079130 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 28 00:41:53.079130 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:41:53.079130 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:41:53.079130 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:41:53.079130 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:41:53.079130 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:41:53.079130 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:41:53.079130 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:41:53.079130 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:41:53.079130 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:41:53.079130 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:41:53.079130 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:41:53.079130 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:41:53.080306 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:41:53.080306 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:41:53.080306 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:41:53.080306 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:41:53.080306 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 28 00:41:53.080306 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 28 00:41:53.080306 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 28 00:41:53.080306 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 28 00:41:53.080306 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 28 00:41:53.080306 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 28 00:41:53.080306 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 28 00:41:53.080306 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:41:53.080902 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ea7d4b5-af8b-4f1f-9f5f-cd85d66ef78d None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:53.081615 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 243/495] 10.222.0.42 () {56 vars in 1102 bytes} [Sun Aug 28 00:41:52 2022] GET /identity/v3/roles/service => generated 85 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:41:53.081615 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dd97fc1-148b-4cd4-99b1-6dcaaf7bbcc2 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:53.121254 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:53.127790 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20997ad3-884d-4304-bf57-0050a7565c0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:53.128438 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20997ad3-884d-4304-bf57-0050a7565c0d None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:53.128713 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20997ad3-884d-4304-bf57-0050a7565c0d None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:53.130995 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20997ad3-884d-4304-bf57-0050a7565c0d None admin] PATH_INFO: `/v3/users/glance` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:53.138022 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20997ad3-884d-4304-bf57-0050a7565c0d None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:53.139479 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20997ad3-884d-4304-bf57-0050a7565c0d None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:53.153063 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-20997ad3-884d-4304-bf57-0050a7565c0d None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Aug 28 00:41:53.153063 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:41:53.153063 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:41:53.153063 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:41:53.153063 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:41:53.153063 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:41:53.153063 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:41:53.153063 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:41:53.153063 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:41:53.153063 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:41:53.153063 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:41:53.153063 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:41:53.153063 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 28 00:41:53.153063 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 28 00:41:53.153063 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 28 00:41:53.153063 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 28 00:41:53.154349 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:41:53.154349 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:41:53.154349 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 28 00:41:53.154349 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 28 00:41:53.154349 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 28 00:41:53.154349 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 28 00:41:53.154349 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:41:53.154349 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:41:53.154349 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:41:53.154349 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:41:53.154349 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:41:53.154349 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:41:53.154349 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:41:53.154349 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:41:53.154349 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:41:53.154349 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:41:53.154349 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:41:53.154349 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:41:53.155560 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:41:53.155560 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:41:53.155560 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 28 00:41:53.155560 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 28 00:41:53.155560 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 28 00:41:53.155560 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 28 00:41:53.155560 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 28 00:41:53.155560 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 28 00:41:53.155560 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Aug 28 00:41:53.155560 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:41:53.156852 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 253/496] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Aug 28 00:41:52 2022] GET /identity/v3/users/glance => generated 84 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:41:53.156922 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20997ad3-884d-4304-bf57-0050a7565c0d None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:53.221982 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dd97fc1-148b-4cd4-99b1-6dcaaf7bbcc2 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:53.247293 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:53.264905 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6984a63f-3ffc-48f9-b7c7-763ddb85d550 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:53.265678 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6984a63f-3ffc-48f9-b7c7-763ddb85d550 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:53.265955 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6984a63f-3ffc-48f9-b7c7-763ddb85d550 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:53.266246 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6984a63f-3ffc-48f9-b7c7-763ddb85d550 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:53.279950 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6984a63f-3ffc-48f9-b7c7-763ddb85d550 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 28 00:41:53.280144 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20997ad3-884d-4304-bf57-0050a7565c0d None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:53.284710 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-6984a63f-3ffc-48f9-b7c7-763ddb85d550 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 28 00:41:53.284710 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:41:53.284710 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:41:53.284710 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:41:53.284710 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:41:53.284710 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:41:53.284710 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:41:53.284710 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:41:53.284710 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:41:53.284710 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:41:53.284710 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:41:53.284710 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:41:53.284710 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 28 00:41:53.284710 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 28 00:41:53.284710 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 28 00:41:53.284710 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 28 00:41:53.285642 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 28 00:41:53.285642 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 28 00:41:53.285642 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 28 00:41:53.285642 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 28 00:41:53.285642 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:41:53.285642 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:41:53.285642 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:41:53.285642 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:41:53.285642 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:41:53.285642 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:41:53.285642 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:41:53.285642 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:41:53.285642 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:41:53.285642 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:41:53.285642 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:41:53.285642 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:41:53.291655 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:41:53.291655 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:41:53.291655 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:41:53.291655 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:41:53.291655 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 28 00:41:53.291655 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 28 00:41:53.291655 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 28 00:41:53.291655 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 28 00:41:53.291655 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 28 00:41:53.291655 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 28 00:41:53.291655 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 28 00:41:53.291655 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:41:53.291655 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 244/497] 10.222.0.42 () {56 vars in 1102 bytes} [Sun Aug 28 00:41:53 2022] GET /identity/v3/roles/service => generated 85 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:41:53.291655 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6984a63f-3ffc-48f9-b7c7-763ddb85d550 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:53.329787 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b95c7500-dc37-49cd-bb76-eb87de98a785 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:53.331162 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b95c7500-dc37-49cd-bb76-eb87de98a785 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:53.331738 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b95c7500-dc37-49cd-bb76-eb87de98a785 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:53.332228 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b95c7500-dc37-49cd-bb76-eb87de98a785 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:53.350218 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b95c7500-dc37-49cd-bb76-eb87de98a785 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:53.352467 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b95c7500-dc37-49cd-bb76-eb87de98a785 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:53.376175 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-b95c7500-dc37-49cd-bb76-eb87de98a785 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:41:53.376175 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:41:53.376175 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:41:53.376175 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:41:53.376175 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:41:53.376175 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:41:53.376175 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:41:53.376175 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:41:53.376175 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:41:53.376175 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:41:53.376175 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:41:53.376175 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:41:53.376175 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 28 00:41:53.376175 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 28 00:41:53.376175 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 28 00:41:53.376175 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 28 00:41:53.378059 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:41:53.378059 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:41:53.378059 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:41:53.378059 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:41:53.378059 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:41:53.378059 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:41:53.378059 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:41:53.378059 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:41:53.378059 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:41:53.378059 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:41:53.378059 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:41:53.378059 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:41:53.378059 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:41:53.378059 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:41:53.378059 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:41:53.378059 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:41:53.378059 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 28 00:41:53.379681 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 28 00:41:53.379681 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:41:53.379681 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:41:53.379681 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 254/498] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:41:53 2022] GET /identity/v3/domains/Default => generated 87 bytes in 222 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:41:53.379681 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b95c7500-dc37-49cd-bb76-eb87de98a785 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:53.436465 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6984a63f-3ffc-48f9-b7c7-763ddb85d550 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:53.443147 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:53.476568 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae0b38bb-8d31-4ba6-a470-1405358596b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:53.477126 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae0b38bb-8d31-4ba6-a470-1405358596b3 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:53.477427 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae0b38bb-8d31-4ba6-a470-1405358596b3 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:53.477687 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae0b38bb-8d31-4ba6-a470-1405358596b3 None admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:53.478352 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae0b38bb-8d31-4ba6-a470-1405358596b3 None admin] RBAC: Adding query filter params (name=service) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:53.478654 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae0b38bb-8d31-4ba6-a470-1405358596b3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:53.480060 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae0b38bb-8d31-4ba6-a470-1405358596b3 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:53.490316 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 245/499] 10.222.0.42 () {56 vars in 1103 bytes} [Sun Aug 28 00:41:53 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:53.491154 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae0b38bb-8d31-4ba6-a470-1405358596b3 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:53.508541 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b95c7500-dc37-49cd-bb76-eb87de98a785 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:53.540742 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:53.544304 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-634c6ac0-7ad1-41db-8ea0-79b53638fd11 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:53.545693 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-634c6ac0-7ad1-41db-8ea0-79b53638fd11 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:53.545693 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-634c6ac0-7ad1-41db-8ea0-79b53638fd11 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:53.545693 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-634c6ac0-7ad1-41db-8ea0-79b53638fd11 None admin] PATH_INFO: `/v3/users` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:53.546588 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-634c6ac0-7ad1-41db-8ea0-79b53638fd11 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:53.546588 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-634c6ac0-7ad1-41db-8ea0-79b53638fd11 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:53.547974 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-634c6ac0-7ad1-41db-8ea0-79b53638fd11 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:53.587044 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 255/500] 10.222.0.42 () {56 vars in 1137 bytes} [Sun Aug 28 00:41:53 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:53.587044 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-634c6ac0-7ad1-41db-8ea0-79b53638fd11 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:53.602907 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae0b38bb-8d31-4ba6-a470-1405358596b3 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:53.642146 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4140e501-5d01-487e-8f98-69a8cca756fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:53.642755 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4140e501-5d01-487e-8f98-69a8cca756fc None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:53.643030 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4140e501-5d01-487e-8f98-69a8cca756fc None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:53.643296 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4140e501-5d01-487e-8f98-69a8cca756fc None admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:53.643957 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4140e501-5d01-487e-8f98-69a8cca756fc None admin] RBAC: Adding query filter params (name=service) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:53.644372 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4140e501-5d01-487e-8f98-69a8cca756fc None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:53.645700 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4140e501-5d01-487e-8f98-69a8cca756fc None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:53.654453 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 246/501] 10.222.0.42 () {56 vars in 1103 bytes} [Sun Aug 28 00:41:53 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:53.655778 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82a90611-ba14-4a33-8546-25e4ab7b8bb4 None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:53.656061 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82a90611-ba14-4a33-8546-25e4ab7b8bb4 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:53.656330 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82a90611-ba14-4a33-8546-25e4ab7b8bb4 None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:53.661077 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 247/502] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:41:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:41:53.661077 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82a90611-ba14-4a33-8546-25e4ab7b8bb4 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:53.705518 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-634c6ac0-7ad1-41db-8ea0-79b53638fd11 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:53.707103 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:53.741038 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fd21603-b6a7-4c6a-ab3d-5c6b1bb378f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:53.741543 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fd21603-b6a7-4c6a-ab3d-5c6b1bb378f1 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:53.741822 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fd21603-b6a7-4c6a-ab3d-5c6b1bb378f1 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:53.742094 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fd21603-b6a7-4c6a-ab3d-5c6b1bb378f1 None admin] PATH_INFO: `/v3/domains` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:53.742878 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fd21603-b6a7-4c6a-ab3d-5c6b1bb378f1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:53.743027 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fd21603-b6a7-4c6a-ab3d-5c6b1bb378f1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:53.744428 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fd21603-b6a7-4c6a-ab3d-5c6b1bb378f1 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:53.756454 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 256/503] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Aug 28 00:41:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:53.757363 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fd21603-b6a7-4c6a-ab3d-5c6b1bb378f1 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:53.775123 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82a90611-ba14-4a33-8546-25e4ab7b8bb4 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:53.782473 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:53.810617 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b73b996-92e7-433c-82b9-d66003f41f69 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:53.811249 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b73b996-92e7-433c-82b9-d66003f41f69 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:53.811522 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b73b996-92e7-433c-82b9-d66003f41f69 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:53.811828 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b73b996-92e7-433c-82b9-d66003f41f69 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:53.822611 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b73b996-92e7-433c-82b9-d66003f41f69 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:53.824068 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b73b996-92e7-433c-82b9-d66003f41f69 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:53.839899 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-4b73b996-92e7-433c-82b9-d66003f41f69 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:41:53.839899 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:41:53.839899 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:41:53.839899 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:41:53.839899 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:41:53.839899 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:41:53.839899 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:41:53.839899 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:41:53.839899 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:41:53.839899 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:41:53.839899 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:41:53.839899 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:41:53.839899 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 28 00:41:53.839899 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 28 00:41:53.839899 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 28 00:41:53.839899 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 28 00:41:53.840849 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:41:53.840849 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:41:53.840849 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:41:53.840849 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:41:53.840849 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:41:53.840849 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:41:53.840849 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:41:53.840849 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:41:53.840849 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:41:53.840849 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:41:53.840849 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:41:53.840849 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:41:53.840849 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:41:53.840849 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:41:53.840849 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:41:53.840849 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:41:53.840849 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 28 00:41:53.841741 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 28 00:41:53.841741 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:41:53.841741 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:41:53.841741 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 248/504] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:41:53 2022] GET /identity/v3/domains/Default => generated 87 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:41:53.841954 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b73b996-92e7-433c-82b9-d66003f41f69 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:53.870070 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fd21603-b6a7-4c6a-ab3d-5c6b1bb378f1 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:53.903846 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcc2a649-b922-4ffd-b19b-67e401fd9bad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:53.904450 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcc2a649-b922-4ffd-b19b-67e401fd9bad None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:53.904720 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcc2a649-b922-4ffd-b19b-67e401fd9bad None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:53.905021 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcc2a649-b922-4ffd-b19b-67e401fd9bad None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:53.937120 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:53.955807 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b73b996-92e7-433c-82b9-d66003f41f69 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:53.978083 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcc2a649-b922-4ffd-b19b-67e401fd9bad None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:53.979945 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcc2a649-b922-4ffd-b19b-67e401fd9bad None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:53.989397 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0152b7bc-76bc-4923-9866-73fd9af64e3c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:53.989673 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0152b7bc-76bc-4923-9866-73fd9af64e3c None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:53.989947 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0152b7bc-76bc-4923-9866-73fd9af64e3c None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:53.990210 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0152b7bc-76bc-4923-9866-73fd9af64e3c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:54.002342 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0152b7bc-76bc-4923-9866-73fd9af64e3c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:54.003721 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0152b7bc-76bc-4923-9866-73fd9af64e3c None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:54.019344 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-0152b7bc-76bc-4923-9866-73fd9af64e3c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:41:54.019344 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:41:54.019344 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:41:54.019344 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:41:54.019344 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:41:54.019344 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:41:54.019344 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:41:54.019344 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:41:54.019344 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:41:54.019344 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:41:54.019344 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:41:54.019344 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:41:54.019344 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 28 00:41:54.019344 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 28 00:41:54.019344 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 28 00:41:54.019344 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 28 00:41:54.020327 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:41:54.020327 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:41:54.020327 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:41:54.020327 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:41:54.020327 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:41:54.020327 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:41:54.020327 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:41:54.020327 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:41:54.020327 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:41:54.020327 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:41:54.020327 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:41:54.020327 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:41:54.020327 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:41:54.020327 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:41:54.020327 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:41:54.020327 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:41:54.020327 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 28 00:41:54.021737 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 28 00:41:54.021737 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:41:54.021737 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:41:54.021737 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 249/505] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:41:53 2022] GET /identity/v3/domains/Default => generated 87 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:41:54.021737 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46179737-922f-4438-99e8-ca794cbc8b5e None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:54.022033 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46179737-922f-4438-99e8-ca794cbc8b5e None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:54.022207 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46179737-922f-4438-99e8-ca794cbc8b5e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:54.047032 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-46179737-922f-4438-99e8-ca794cbc8b5e None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:41:54.048815 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-46179737-922f-4438-99e8-ca794cbc8b5e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:41:54.054108 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:54.108493 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 257/506] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Aug 28 00:41:53 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 351 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 28 00:41:54.109893 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcc2a649-b922-4ffd-b19b-67e401fd9bad None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:54.143846 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 250/507] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:41:54 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 28 00:41:54.144721 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46179737-922f-4438-99e8-ca794cbc8b5e None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:54.182025 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:54.229174 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcc2a649-b922-4ffd-b19b-67e401fd9bad None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:54.258175 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46179737-922f-4438-99e8-ca794cbc8b5e None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:54.270665 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-652eed50-09c5-4697-b3c4-c609661135bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:54.271650 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-652eed50-09c5-4697-b3c4-c609661135bb None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:54.271967 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-652eed50-09c5-4697-b3c4-c609661135bb None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:54.272335 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-652eed50-09c5-4697-b3c4-c609661135bb None admin] PATH_INFO: `/v3/users/nova` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:54.280658 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-652eed50-09c5-4697-b3c4-c609661135bb None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:54.282297 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-652eed50-09c5-4697-b3c4-c609661135bb None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:54.294351 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-652eed50-09c5-4697-b3c4-c609661135bb None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 28 00:41:54.294351 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:41:54.294351 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:41:54.294351 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:41:54.294351 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:41:54.294351 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:41:54.294351 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:41:54.294351 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:41:54.294351 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:41:54.294351 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:41:54.294351 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:41:54.294351 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:41:54.294351 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 28 00:41:54.294351 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 28 00:41:54.294351 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 28 00:41:54.294351 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 28 00:41:54.295278 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:41:54.295278 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:41:54.295278 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 28 00:41:54.295278 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 28 00:41:54.295278 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 28 00:41:54.295278 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 28 00:41:54.295278 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:41:54.295278 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:41:54.295278 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:41:54.295278 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:41:54.295278 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:41:54.295278 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:41:54.295278 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:41:54.295278 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:41:54.295278 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:41:54.295278 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:41:54.295278 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:41:54.295278 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:41:54.296210 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:41:54.296210 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:41:54.296210 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 28 00:41:54.296210 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 28 00:41:54.296210 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 28 00:41:54.296210 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 28 00:41:54.296210 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 28 00:41:54.296210 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 28 00:41:54.296210 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 28 00:41:54.296210 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:41:54.296210 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 258/508] 10.222.0.42 () {56 vars in 1093 bytes} [Sun Aug 28 00:41:54 2022] GET /identity/v3/users/nova => generated 82 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:41:54.296775 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-652eed50-09c5-4697-b3c4-c609661135bb None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:54.308739 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b79ca49-7b76-486d-b728-ce221b82ee9f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:54.309676 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b79ca49-7b76-486d-b728-ce221b82ee9f None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:54.310596 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b79ca49-7b76-486d-b728-ce221b82ee9f None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:54.311038 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b79ca49-7b76-486d-b728-ce221b82ee9f None admin] PATH_INFO: `/v3/domains` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:54.312351 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b79ca49-7b76-486d-b728-ce221b82ee9f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:54.313140 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b79ca49-7b76-486d-b728-ce221b82ee9f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:54.315015 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b79ca49-7b76-486d-b728-ce221b82ee9f None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:54.327195 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:54.335246 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 251/509] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Aug 28 00:41:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:54.335246 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b79ca49-7b76-486d-b728-ce221b82ee9f None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:54.429648 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-652eed50-09c5-4697-b3c4-c609661135bb None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:54.444140 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b79ca49-7b76-486d-b728-ce221b82ee9f None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:54.453471 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:54.466340 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5976a391-738a-4d6a-9bc8-ad638bdcb912 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:54.467148 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5976a391-738a-4d6a-9bc8-ad638bdcb912 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:54.467283 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5976a391-738a-4d6a-9bc8-ad638bdcb912 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:54.467498 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5976a391-738a-4d6a-9bc8-ad638bdcb912 None admin] PATH_INFO: `/v3/domains` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:54.468202 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5976a391-738a-4d6a-9bc8-ad638bdcb912 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:54.468539 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5976a391-738a-4d6a-9bc8-ad638bdcb912 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:54.469968 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5976a391-738a-4d6a-9bc8-ad638bdcb912 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:54.478594 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e131eff7-ac56-4f16-959f-789244cdee67 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:54.479264 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e131eff7-ac56-4f16-959f-789244cdee67 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:54.479537 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e131eff7-ac56-4f16-959f-789244cdee67 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:54.479768 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e131eff7-ac56-4f16-959f-789244cdee67 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:54.482982 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 259/510] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Aug 28 00:41:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:54.484190 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5976a391-738a-4d6a-9bc8-ad638bdcb912 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:54.491781 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e131eff7-ac56-4f16-959f-789244cdee67 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:54.493322 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e131eff7-ac56-4f16-959f-789244cdee67 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:54.508436 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-e131eff7-ac56-4f16-959f-789244cdee67 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:41:54.508436 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:41:54.508436 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:41:54.508436 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:41:54.508436 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:41:54.508436 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:41:54.508436 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:41:54.508436 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:41:54.508436 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:41:54.508436 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:41:54.508436 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:41:54.508436 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:41:54.508436 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 28 00:41:54.508436 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 28 00:41:54.508436 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 28 00:41:54.508436 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 28 00:41:54.509469 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:41:54.509469 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:41:54.509469 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:41:54.509469 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:41:54.509469 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:41:54.509469 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:41:54.509469 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:41:54.509469 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:41:54.509469 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:41:54.509469 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:41:54.509469 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:41:54.509469 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:41:54.509469 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:41:54.509469 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:41:54.509469 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:41:54.509469 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:41:54.509469 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 28 00:41:54.510313 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 28 00:41:54.510313 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:41:54.510313 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:41:54.510313 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 252/511] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:41:54 2022] GET /identity/v3/domains/Default => generated 87 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:41:54.510313 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e131eff7-ac56-4f16-959f-789244cdee67 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:54.567606 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:54.595867 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5976a391-738a-4d6a-9bc8-ad638bdcb912 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:54.617203 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e131eff7-ac56-4f16-959f-789244cdee67 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:54.637519 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c711a15d-06a4-4686-83e2-286b04379bfa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:54.638119 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c711a15d-06a4-4686-83e2-286b04379bfa None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:54.638394 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c711a15d-06a4-4686-83e2-286b04379bfa None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:54.638664 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c711a15d-06a4-4686-83e2-286b04379bfa None admin] PATH_INFO: `/v3/users` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:54.639393 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c711a15d-06a4-4686-83e2-286b04379bfa None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:54.639694 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c711a15d-06a4-4686-83e2-286b04379bfa None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:54.641252 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c711a15d-06a4-4686-83e2-286b04379bfa None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:54.652162 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-033f0f2d-4468-479c-a539-361766f2a22f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:54.652802 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-033f0f2d-4468-479c-a539-361766f2a22f None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:54.653123 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-033f0f2d-4468-479c-a539-361766f2a22f None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:54.653397 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-033f0f2d-4468-479c-a539-361766f2a22f None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:54.659441 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 260/512] 10.222.0.42 () {56 vars in 1133 bytes} [Sun Aug 28 00:41:54 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:54.660316 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c711a15d-06a4-4686-83e2-286b04379bfa None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:54.665224 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-033f0f2d-4468-479c-a539-361766f2a22f None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:54.666711 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-033f0f2d-4468-479c-a539-361766f2a22f None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:54.680176 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-033f0f2d-4468-479c-a539-361766f2a22f None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Aug 28 00:41:54.680176 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:41:54.680176 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:41:54.680176 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:41:54.680176 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:41:54.680176 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:41:54.680176 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:41:54.680176 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:41:54.680176 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:41:54.680176 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:41:54.680176 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:41:54.680176 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:41:54.680176 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 28 00:41:54.680176 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 28 00:41:54.680176 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 28 00:41:54.680176 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 28 00:41:54.681340 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:41:54.681340 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:41:54.681340 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 28 00:41:54.681340 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 28 00:41:54.681340 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 28 00:41:54.681340 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 28 00:41:54.681340 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:41:54.681340 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:41:54.681340 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:41:54.681340 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:41:54.681340 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:41:54.681340 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:41:54.681340 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:41:54.681340 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:41:54.681340 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:41:54.681340 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:41:54.681340 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:41:54.681340 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:41:54.682405 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:41:54.682405 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:41:54.682405 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 28 00:41:54.682405 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 28 00:41:54.682405 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 28 00:41:54.682405 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 28 00:41:54.682405 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 28 00:41:54.682405 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 28 00:41:54.682405 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Aug 28 00:41:54.682405 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:41:54.682405 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 253/513] 10.222.0.42 () {56 vars in 1102 bytes} [Sun Aug 28 00:41:54 2022] GET /identity/v3/users/neutron => generated 85 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:41:54.683153 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-033f0f2d-4468-479c-a539-361766f2a22f None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:54.711805 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:54.772631 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c711a15d-06a4-4686-83e2-286b04379bfa None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:54.793138 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-033f0f2d-4468-479c-a539-361766f2a22f None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:54.807077 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7abc6596-e214-444a-9f2f-d097af7f408e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:54.807339 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:54.807701 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7abc6596-e214-444a-9f2f-d097af7f408e None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:54.807972 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7abc6596-e214-444a-9f2f-d097af7f408e None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:54.808244 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7abc6596-e214-444a-9f2f-d097af7f408e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:54.820509 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7abc6596-e214-444a-9f2f-d097af7f408e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:54.822036 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7abc6596-e214-444a-9f2f-d097af7f408e None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:54.831044 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9265735-1832-4b52-bf32-9bf5dd664aa2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:54.831695 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9265735-1832-4b52-bf32-9bf5dd664aa2 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:54.831977 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9265735-1832-4b52-bf32-9bf5dd664aa2 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:54.832253 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9265735-1832-4b52-bf32-9bf5dd664aa2 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:54.837471 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-7abc6596-e214-444a-9f2f-d097af7f408e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:41:54.837471 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:41:54.837471 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:41:54.837471 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:41:54.837471 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:41:54.837471 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:41:54.837471 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:41:54.837471 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:41:54.837471 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:41:54.837471 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:41:54.837471 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:41:54.837471 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:41:54.837471 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 28 00:41:54.837471 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 28 00:41:54.837471 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 28 00:41:54.837471 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 28 00:41:54.838592 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:41:54.838592 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:41:54.838592 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:41:54.838592 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:41:54.838592 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:41:54.838592 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:41:54.838592 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:41:54.838592 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:41:54.838592 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:41:54.838592 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:41:54.838592 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:41:54.838592 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:41:54.838592 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:41:54.838592 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:41:54.838592 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:41:54.838592 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:41:54.838592 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 28 00:41:54.839427 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 28 00:41:54.839427 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:41:54.839427 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:41:54.839427 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 261/514] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:41:54 2022] GET /identity/v3/domains/Default => generated 87 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:41:54.839427 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7abc6596-e214-444a-9f2f-d097af7f408e None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:54.844510 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9265735-1832-4b52-bf32-9bf5dd664aa2 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:54.846083 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9265735-1832-4b52-bf32-9bf5dd664aa2 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:54.858195 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-e9265735-1832-4b52-bf32-9bf5dd664aa2 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Aug 28 00:41:54.858195 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:41:54.858195 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:41:54.858195 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:41:54.858195 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:41:54.858195 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:41:54.858195 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:41:54.858195 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:41:54.858195 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:41:54.858195 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:41:54.858195 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:41:54.858195 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:41:54.858195 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 28 00:41:54.858195 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 28 00:41:54.858195 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 28 00:41:54.858195 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 28 00:41:54.859336 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:41:54.859336 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:41:54.859336 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 28 00:41:54.859336 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 28 00:41:54.859336 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 28 00:41:54.859336 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 28 00:41:54.859336 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:41:54.859336 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:41:54.859336 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:41:54.859336 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:41:54.859336 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:41:54.859336 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:41:54.859336 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:41:54.859336 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:41:54.859336 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:41:54.859336 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:41:54.859336 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:41:54.859336 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:41:54.860420 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:41:54.860420 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:41:54.860420 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 28 00:41:54.860420 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 28 00:41:54.860420 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 28 00:41:54.860420 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 28 00:41:54.860420 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 28 00:41:54.860420 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 28 00:41:54.860420 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Aug 28 00:41:54.860420 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:41:54.860420 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 254/515] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Aug 28 00:41:54 2022] GET /identity/v3/users/cinder => generated 84 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:41:54.860420 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9265735-1832-4b52-bf32-9bf5dd664aa2 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:54.947252 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:54.957422 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7abc6596-e214-444a-9f2f-d097af7f408e None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:54.968474 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9265735-1832-4b52-bf32-9bf5dd664aa2 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:54.995509 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23c9be01-b891-42ca-93f2-929911fee919 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:54.996123 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23c9be01-b891-42ca-93f2-929911fee919 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:54.996421 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23c9be01-b891-42ca-93f2-929911fee919 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:54.996694 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23c9be01-b891-42ca-93f2-929911fee919 None admin] PATH_INFO: `/v3/domains` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:54.997372 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23c9be01-b891-42ca-93f2-929911fee919 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:54.997675 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23c9be01-b891-42ca-93f2-929911fee919 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:54.999064 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23c9be01-b891-42ca-93f2-929911fee919 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:55.006244 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-655de081-d6ad-4306-bd74-26fd924284f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:55.006847 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-655de081-d6ad-4306-bd74-26fd924284f9 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:55.007123 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-655de081-d6ad-4306-bd74-26fd924284f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:55.007390 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-655de081-d6ad-4306-bd74-26fd924284f9 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:55.011057 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 262/516] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Aug 28 00:41:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:55.011983 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23c9be01-b891-42ca-93f2-929911fee919 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:55.063240 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:55.092423 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-655de081-d6ad-4306-bd74-26fd924284f9 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:55.094344 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-655de081-d6ad-4306-bd74-26fd924284f9 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:55.130151 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23c9be01-b891-42ca-93f2-929911fee919 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:55.165172 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15bbed2c-98fe-4f3f-8768-7ae354b42361 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:55.165756 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15bbed2c-98fe-4f3f-8768-7ae354b42361 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:55.166033 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15bbed2c-98fe-4f3f-8768-7ae354b42361 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:55.166301 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15bbed2c-98fe-4f3f-8768-7ae354b42361 None admin] PATH_INFO: `/v3/users` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:55.167029 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15bbed2c-98fe-4f3f-8768-7ae354b42361 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:55.167322 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15bbed2c-98fe-4f3f-8768-7ae354b42361 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:55.168751 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15bbed2c-98fe-4f3f-8768-7ae354b42361 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:55.190982 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 263/517] 10.222.0.42 () {56 vars in 1139 bytes} [Sun Aug 28 00:41:55 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:55.193431 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15bbed2c-98fe-4f3f-8768-7ae354b42361 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:55.206089 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:55.210854 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 255/518] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Aug 28 00:41:54 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 351 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 28 00:41:55.214295 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-655de081-d6ad-4306-bd74-26fd924284f9 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:55.290115 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:55.294163 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15bbed2c-98fe-4f3f-8768-7ae354b42361 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:55.321999 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-655de081-d6ad-4306-bd74-26fd924284f9 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:55.330623 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71fce992-c11b-4d5e-8c30-6c3d3e7c58a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:55.331240 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71fce992-c11b-4d5e-8c30-6c3d3e7c58a2 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:55.331513 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71fce992-c11b-4d5e-8c30-6c3d3e7c58a2 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:55.331771 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71fce992-c11b-4d5e-8c30-6c3d3e7c58a2 None admin] PATH_INFO: `/v3/domains` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:55.332441 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71fce992-c11b-4d5e-8c30-6c3d3e7c58a2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:55.332713 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71fce992-c11b-4d5e-8c30-6c3d3e7c58a2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:55.334119 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71fce992-c11b-4d5e-8c30-6c3d3e7c58a2 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:55.347980 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 264/519] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Aug 28 00:41:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:55.348847 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71fce992-c11b-4d5e-8c30-6c3d3e7c58a2 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:55.365778 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bb8833b-6274-4a80-8c60-0122734bbb66 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:55.366373 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bb8833b-6274-4a80-8c60-0122734bbb66 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:55.366654 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bb8833b-6274-4a80-8c60-0122734bbb66 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:55.367117 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bb8833b-6274-4a80-8c60-0122734bbb66 None admin] PATH_INFO: `/v3/users` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:55.367637 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bb8833b-6274-4a80-8c60-0122734bbb66 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:55.368184 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bb8833b-6274-4a80-8c60-0122734bbb66 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:55.369401 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bb8833b-6274-4a80-8c60-0122734bbb66 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:55.405619 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 256/520] 10.222.0.42 () {56 vars in 1137 bytes} [Sun Aug 28 00:41:55 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:55.406481 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bb8833b-6274-4a80-8c60-0122734bbb66 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:55.426035 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:55.459125 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71fce992-c11b-4d5e-8c30-6c3d3e7c58a2 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:55.497935 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7b2f2ad-e3fd-4617-b813-3abe25394bf9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:55.498572 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7b2f2ad-e3fd-4617-b813-3abe25394bf9 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:55.498846 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7b2f2ad-e3fd-4617-b813-3abe25394bf9 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:55.499114 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7b2f2ad-e3fd-4617-b813-3abe25394bf9 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:55.506179 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7b2f2ad-e3fd-4617-b813-3abe25394bf9 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:55.507628 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7b2f2ad-e3fd-4617-b813-3abe25394bf9 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:55.508098 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bb8833b-6274-4a80-8c60-0122734bbb66 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:55.519703 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-e7b2f2ad-e3fd-4617-b813-3abe25394bf9 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 28 00:41:55.519703 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:41:55.519703 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:41:55.519703 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:41:55.519703 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:41:55.519703 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:41:55.519703 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:41:55.519703 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:41:55.519703 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:41:55.519703 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:41:55.519703 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:41:55.519703 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:41:55.519703 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 28 00:41:55.519703 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 28 00:41:55.519703 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 28 00:41:55.520625 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 28 00:41:55.520625 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:41:55.520625 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:41:55.520625 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:41:55.520625 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:41:55.520625 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:41:55.520625 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:41:55.520625 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:41:55.520625 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:41:55.520625 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:41:55.520625 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:41:55.520625 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:41:55.520625 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:41:55.520625 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:41:55.520625 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:41:55.520625 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:41:55.520625 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:41:55.521507 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 28 00:41:55.521507 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 28 00:41:55.521507 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 28 00:41:55.521507 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 28 00:41:55.521507 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 28 00:41:55.521507 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 28 00:41:55.521507 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 28 00:41:55.521507 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:41:55.521507 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 265/521] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Aug 28 00:41:55 2022] GET /identity/v3/projects/service => generated 88 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:41:55.522037 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7b2f2ad-e3fd-4617-b813-3abe25394bf9 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:55.539654 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:55.550234 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1c18158-97b0-4a69-a748-d8928788f98d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:55.550854 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1c18158-97b0-4a69-a748-d8928788f98d None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:55.551124 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1c18158-97b0-4a69-a748-d8928788f98d None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:55.551397 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1c18158-97b0-4a69-a748-d8928788f98d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:55.620221 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1c18158-97b0-4a69-a748-d8928788f98d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:55.621662 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1c18158-97b0-4a69-a748-d8928788f98d None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:55.639301 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7b2f2ad-e3fd-4617-b813-3abe25394bf9 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:55.668743 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:55.678026 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10690157-82df-4730-b15e-ec3a9d7bbe8e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:55.678663 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10690157-82df-4730-b15e-ec3a9d7bbe8e None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:55.678927 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10690157-82df-4730-b15e-ec3a9d7bbe8e None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:55.679193 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10690157-82df-4730-b15e-ec3a9d7bbe8e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:55.690841 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10690157-82df-4730-b15e-ec3a9d7bbe8e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:55.692301 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10690157-82df-4730-b15e-ec3a9d7bbe8e None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:55.706909 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-10690157-82df-4730-b15e-ec3a9d7bbe8e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:41:55.706909 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:41:55.706909 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:41:55.706909 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:41:55.706909 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:41:55.706909 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:41:55.706909 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:41:55.706909 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:41:55.706909 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:41:55.706909 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:41:55.706909 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:41:55.706909 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:41:55.706909 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 28 00:41:55.706909 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 28 00:41:55.706909 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 28 00:41:55.706909 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 28 00:41:55.707980 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:41:55.707980 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:41:55.707980 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:41:55.707980 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:41:55.707980 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:41:55.707980 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:41:55.707980 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:41:55.707980 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:41:55.707980 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:41:55.707980 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:41:55.707980 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:41:55.707980 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:41:55.707980 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:41:55.707980 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:41:55.707980 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:41:55.707980 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:41:55.707980 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 28 00:41:55.709127 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 28 00:41:55.709127 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:41:55.709127 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:41:55.709127 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 266/522] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:41:55 2022] GET /identity/v3/domains/Default => generated 87 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:41:55.709127 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10690157-82df-4730-b15e-ec3a9d7bbe8e None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:55.734744 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 257/523] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Aug 28 00:41:55 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 329 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 28 00:41:55.735704 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1c18158-97b0-4a69-a748-d8928788f98d None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:55.767054 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:55.811866 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10690157-82df-4730-b15e-ec3a9d7bbe8e None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:55.841963 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1c18158-97b0-4a69-a748-d8928788f98d None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:55.846658 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3059bb48-101f-429a-b900-da67c6d40b51 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:55.847263 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3059bb48-101f-429a-b900-da67c6d40b51 None admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:55.847542 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3059bb48-101f-429a-b900-da67c6d40b51 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:55.847812 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3059bb48-101f-429a-b900-da67c6d40b51 None admin] PATH_INFO: `/v3/users` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:55.848458 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3059bb48-101f-429a-b900-da67c6d40b51 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:55.849942 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3059bb48-101f-429a-b900-da67c6d40b51 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:55.876594 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 267/524] 10.222.0.42 () {60 vars in 1132 bytes} [Sun Aug 28 00:41:55 2022] POST /identity/v3/users => generated 247 bytes in 162 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:41:55.876594 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3059bb48-101f-429a-b900-da67c6d40b51 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:55.880003 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43d71b51-6bb2-47a2-811c-73c5ce664b77 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:55.880614 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d71b51-6bb2-47a2-811c-73c5ce664b77 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:55.881036 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d71b51-6bb2-47a2-811c-73c5ce664b77 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:55.881206 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d71b51-6bb2-47a2-811c-73c5ce664b77 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:55.941482 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:55.959973 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43d71b51-6bb2-47a2-811c-73c5ce664b77 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:55.961794 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43d71b51-6bb2-47a2-811c-73c5ce664b77 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:55.997124 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:56.042603 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3059bb48-101f-429a-b900-da67c6d40b51 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:56.082841 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 258/525] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Aug 28 00:41:55 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 348 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 28 00:41:56.083796 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43d71b51-6bb2-47a2-811c-73c5ce664b77 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:56.091743 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f42a79a6-3509-4122-8061-9543a1ae4bd5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:56.092388 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f42a79a6-3509-4122-8061-9543a1ae4bd5 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:56.092649 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f42a79a6-3509-4122-8061-9543a1ae4bd5 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:56.092922 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f42a79a6-3509-4122-8061-9543a1ae4bd5 None admin] PATH_INFO: `/v3/projects` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:56.093716 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f42a79a6-3509-4122-8061-9543a1ae4bd5 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:56.094026 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f42a79a6-3509-4122-8061-9543a1ae4bd5 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:56.095541 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f42a79a6-3509-4122-8061-9543a1ae4bd5 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:56.121292 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 268/526] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Aug 28 00:41:55 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 250 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:56.122389 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f42a79a6-3509-4122-8061-9543a1ae4bd5 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:56.187929 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43d71b51-6bb2-47a2-811c-73c5ce664b77 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:56.222489 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:56.229044 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59e51408-7c3e-4178-b158-1f1f1e037d4b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:56.229707 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e51408-7c3e-4178-b158-1f1f1e037d4b None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:56.229979 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e51408-7c3e-4178-b158-1f1f1e037d4b None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:56.230250 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e51408-7c3e-4178-b158-1f1f1e037d4b None admin] PATH_INFO: `/v3/domains` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:56.230940 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59e51408-7c3e-4178-b158-1f1f1e037d4b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:56.231238 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59e51408-7c3e-4178-b158-1f1f1e037d4b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:56.232619 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59e51408-7c3e-4178-b158-1f1f1e037d4b None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:56.233889 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:56.239476 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f42a79a6-3509-4122-8061-9543a1ae4bd5 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:56.245865 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 259/527] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Aug 28 00:41:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:56.246957 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59e51408-7c3e-4178-b158-1f1f1e037d4b None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:56.280188 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a060507-d3a2-4875-aa44-06f6875b6e1e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:56.280882 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a060507-d3a2-4875-aa44-06f6875b6e1e None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:56.281288 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a060507-d3a2-4875-aa44-06f6875b6e1e None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:56.281753 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a060507-d3a2-4875-aa44-06f6875b6e1e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:56.292235 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a060507-d3a2-4875-aa44-06f6875b6e1e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:56.293752 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a060507-d3a2-4875-aa44-06f6875b6e1e None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:56.309964 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-7a060507-d3a2-4875-aa44-06f6875b6e1e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:41:56.309964 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:41:56.309964 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:41:56.309964 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:41:56.309964 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:41:56.309964 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:41:56.309964 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:41:56.309964 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:41:56.309964 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:41:56.309964 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:41:56.309964 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:41:56.309964 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:41:56.309964 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 28 00:41:56.309964 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 28 00:41:56.309964 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 28 00:41:56.309964 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 28 00:41:56.310955 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:41:56.310955 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:41:56.310955 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:41:56.310955 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:41:56.310955 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:41:56.310955 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:41:56.310955 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:41:56.310955 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:41:56.310955 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:41:56.310955 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:41:56.310955 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:41:56.310955 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:41:56.310955 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:41:56.310955 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:41:56.310955 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:41:56.310955 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:41:56.310955 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 28 00:41:56.311830 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 28 00:41:56.311830 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:41:56.311830 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:41:56.311830 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 269/528] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:41:56 2022] GET /identity/v3/domains/Default => generated 87 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:41:56.312057 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a060507-d3a2-4875-aa44-06f6875b6e1e None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:56.366125 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59e51408-7c3e-4178-b158-1f1f1e037d4b None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:56.403873 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0df7cac-d443-4b11-81d3-60179cb73883 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:56.404600 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0df7cac-d443-4b11-81d3-60179cb73883 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:56.404780 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0df7cac-d443-4b11-81d3-60179cb73883 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:56.405076 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0df7cac-d443-4b11-81d3-60179cb73883 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:56.416456 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0df7cac-d443-4b11-81d3-60179cb73883 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:56.417981 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0df7cac-d443-4b11-81d3-60179cb73883 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:56.430675 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a060507-d3a2-4875-aa44-06f6875b6e1e None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:56.434435 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-b0df7cac-d443-4b11-81d3-60179cb73883 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:41:56.434435 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:41:56.434435 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:41:56.434435 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:41:56.434435 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:41:56.434435 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:41:56.434435 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:41:56.434435 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:41:56.434435 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:41:56.434435 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:41:56.434435 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:41:56.434435 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:41:56.434435 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 28 00:41:56.434435 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 28 00:41:56.434435 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 28 00:41:56.434435 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 28 00:41:56.435849 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:41:56.435849 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:41:56.435849 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:41:56.435849 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:41:56.435849 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:41:56.435849 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:41:56.435849 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:41:56.435849 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:41:56.435849 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:41:56.435849 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:41:56.435849 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:41:56.435849 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:41:56.435849 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:41:56.435849 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:41:56.435849 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:41:56.435849 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:41:56.435849 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 28 00:41:56.437862 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 28 00:41:56.437862 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:41:56.437862 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:41:56.437862 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 260/529] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:41:56 2022] GET /identity/v3/domains/Default => generated 87 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:41:56.437862 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0df7cac-d443-4b11-81d3-60179cb73883 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:56.476822 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06a96f83-4ca0-47f5-b926-522e113a7a25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:56.477138 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06a96f83-4ca0-47f5-b926-522e113a7a25 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:56.477576 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06a96f83-4ca0-47f5-b926-522e113a7a25 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:56.477966 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06a96f83-4ca0-47f5-b926-522e113a7a25 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:56.478737 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06a96f83-4ca0-47f5-b926-522e113a7a25 None admin] RBAC: Adding query filter params (role.id=348070fea3dd4cc2a6e00e29fe82cc25, scope.project.id=f53e6fdb369b480fbebb4086a505df67, user.id=14176b54cf6640abb3b1c14e3824d4f1) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:56.479051 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06a96f83-4ca0-47f5-b926-522e113a7a25 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:56.480671 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06a96f83-4ca0-47f5-b926-522e113a7a25 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:56.487040 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 270/530] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Aug 28 00:41:56 2022] GET /identity/v3/role_assignments?user.id=14176b54cf6640abb3b1c14e3824d4f1&scope.project.id=f53e6fdb369b480fbebb4086a505df67&role.id=348070fea3dd4cc2a6e00e29fe82cc25 => generated 259 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:56.487885 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06a96f83-4ca0-47f5-b926-522e113a7a25 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:56.506128 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:56.522656 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:56.587865 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0df7cac-d443-4b11-81d3-60179cb73883 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:56.632802 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4fb1947-2c6a-4a74-b751-45fe2a478c89 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:56.633482 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4fb1947-2c6a-4a74-b751-45fe2a478c89 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:56.633765 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4fb1947-2c6a-4a74-b751-45fe2a478c89 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:56.634040 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4fb1947-2c6a-4a74-b751-45fe2a478c89 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:56.642002 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4fb1947-2c6a-4a74-b751-45fe2a478c89 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:56.643549 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4fb1947-2c6a-4a74-b751-45fe2a478c89 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:56.652839 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06a96f83-4ca0-47f5-b926-522e113a7a25 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:56.657303 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-c4fb1947-2c6a-4a74-b751-45fe2a478c89 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 28 00:41:56.657303 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:41:56.657303 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:41:56.657303 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:41:56.657303 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:41:56.657303 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:41:56.657303 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:41:56.657303 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:41:56.657303 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:41:56.657303 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:41:56.657303 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:41:56.657303 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:41:56.657303 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 28 00:41:56.657303 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 28 00:41:56.657303 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 28 00:41:56.658211 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 28 00:41:56.658211 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:41:56.658211 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:41:56.658211 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:41:56.658211 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:41:56.658211 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:41:56.658211 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:41:56.658211 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:41:56.658211 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:41:56.658211 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:41:56.658211 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:41:56.658211 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:41:56.658211 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:41:56.658211 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:41:56.658211 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:41:56.658211 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:41:56.658211 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:41:56.659045 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 28 00:41:56.659045 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 28 00:41:56.659045 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 28 00:41:56.659045 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 28 00:41:56.659045 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 28 00:41:56.659045 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 28 00:41:56.659045 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 28 00:41:56.659045 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:41:56.663089 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 261/531] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Aug 28 00:41:56 2022] GET /identity/v3/projects/service => generated 88 bytes in 222 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:41:56.664060 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4fb1947-2c6a-4a74-b751-45fe2a478c89 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:56.710059 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac8a548d-dc63-4ee6-9b53-da52241783e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:56.710673 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac8a548d-dc63-4ee6-9b53-da52241783e2 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:56.710947 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac8a548d-dc63-4ee6-9b53-da52241783e2 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:56.711214 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac8a548d-dc63-4ee6-9b53-da52241783e2 None admin] PATH_INFO: `/v3/domains` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:56.711971 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac8a548d-dc63-4ee6-9b53-da52241783e2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:56.712268 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac8a548d-dc63-4ee6-9b53-da52241783e2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:56.713733 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac8a548d-dc63-4ee6-9b53-da52241783e2 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:56.731027 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 271/532] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Aug 28 00:41:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 243 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:56.731928 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac8a548d-dc63-4ee6-9b53-da52241783e2 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:56.785512 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4fb1947-2c6a-4a74-b751-45fe2a478c89 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:56.787268 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:56.829485 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:56.844436 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f45a5c93-5e6e-498a-ad4c-a3cc7c641602 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:56.845670 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f45a5c93-5e6e-498a-ad4c-a3cc7c641602 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:56.845670 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f45a5c93-5e6e-498a-ad4c-a3cc7c641602 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:56.845845 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f45a5c93-5e6e-498a-ad4c-a3cc7c641602 None admin] PATH_INFO: `/v3/domains` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:56.846389 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f45a5c93-5e6e-498a-ad4c-a3cc7c641602 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:56.846693 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f45a5c93-5e6e-498a-ad4c-a3cc7c641602 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:56.848102 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f45a5c93-5e6e-498a-ad4c-a3cc7c641602 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:56.865751 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac8a548d-dc63-4ee6-9b53-da52241783e2 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:56.869051 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 262/533] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Aug 28 00:41:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:56.871092 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f45a5c93-5e6e-498a-ad4c-a3cc7c641602 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:56.909306 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dae777f-c386-4431-a804-46221349ac61 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:56.910013 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dae777f-c386-4431-a804-46221349ac61 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:56.910382 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dae777f-c386-4431-a804-46221349ac61 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:56.910654 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dae777f-c386-4431-a804-46221349ac61 None admin] PATH_INFO: `/v3/projects` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:56.911348 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dae777f-c386-4431-a804-46221349ac61 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:56.911648 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dae777f-c386-4431-a804-46221349ac61 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:56.913306 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dae777f-c386-4431-a804-46221349ac61 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:56.932174 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 272/534] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Aug 28 00:41:56 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:56.933391 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dae777f-c386-4431-a804-46221349ac61 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:57.010032 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f45a5c93-5e6e-498a-ad4c-a3cc7c641602 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:57.053892 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03f848ce-12ed-4e00-90ac-5ec94e8fbf23 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:57.054572 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03f848ce-12ed-4e00-90ac-5ec94e8fbf23 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:57.054850 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03f848ce-12ed-4e00-90ac-5ec94e8fbf23 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:57.055122 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03f848ce-12ed-4e00-90ac-5ec94e8fbf23 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:57.062693 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03f848ce-12ed-4e00-90ac-5ec94e8fbf23 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:57.064206 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03f848ce-12ed-4e00-90ac-5ec94e8fbf23 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:57.073464 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dae777f-c386-4431-a804-46221349ac61 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:57.076973 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-03f848ce-12ed-4e00-90ac-5ec94e8fbf23 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 28 00:41:57.076973 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:41:57.076973 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:41:57.076973 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:41:57.076973 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:41:57.076973 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:41:57.076973 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:41:57.076973 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:41:57.076973 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:41:57.076973 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:41:57.076973 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:41:57.076973 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:41:57.076973 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 28 00:41:57.076973 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 28 00:41:57.076973 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 28 00:41:57.077878 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 28 00:41:57.077878 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:41:57.077878 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:41:57.077878 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:41:57.077878 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:41:57.077878 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:41:57.077878 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:41:57.077878 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:41:57.077878 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:41:57.077878 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:41:57.077878 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:41:57.077878 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:41:57.077878 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:41:57.077878 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:41:57.077878 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:41:57.077878 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:41:57.077878 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:41:57.078728 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 28 00:41:57.078728 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 28 00:41:57.078728 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 28 00:41:57.078728 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 28 00:41:57.078728 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 28 00:41:57.078728 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 28 00:41:57.078728 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 28 00:41:57.078728 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:41:57.078728 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 263/535] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Aug 28 00:41:56 2022] GET /identity/v3/projects/service => generated 88 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:41:57.079195 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03f848ce-12ed-4e00-90ac-5ec94e8fbf23 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:57.121490 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:57.135503 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-330c165f-0565-4d22-90f5-ebc98b8382cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:57.136537 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-330c165f-0565-4d22-90f5-ebc98b8382cb None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:57.137374 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-330c165f-0565-4d22-90f5-ebc98b8382cb None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:57.137967 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-330c165f-0565-4d22-90f5-ebc98b8382cb None admin] PATH_INFO: `/v3/projects/service` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:57.141235 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:57.150654 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-330c165f-0565-4d22-90f5-ebc98b8382cb None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:57.152931 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-330c165f-0565-4d22-90f5-ebc98b8382cb None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:57.170333 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-330c165f-0565-4d22-90f5-ebc98b8382cb None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 28 00:41:57.170333 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:41:57.170333 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:41:57.170333 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:41:57.170333 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:41:57.170333 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:41:57.170333 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:41:57.170333 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:41:57.170333 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:41:57.170333 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:41:57.170333 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:41:57.170333 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:41:57.170333 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 28 00:41:57.170333 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 28 00:41:57.170333 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 28 00:41:57.171314 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 28 00:41:57.171314 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:41:57.171314 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:41:57.171314 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:41:57.171314 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:41:57.171314 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:41:57.171314 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:41:57.171314 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:41:57.171314 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:41:57.171314 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:41:57.171314 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:41:57.171314 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:41:57.171314 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:41:57.171314 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:41:57.171314 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:41:57.171314 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:41:57.171314 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:41:57.172166 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 28 00:41:57.172166 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 28 00:41:57.172166 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 28 00:41:57.172166 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 28 00:41:57.172166 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 28 00:41:57.172166 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 28 00:41:57.172166 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 28 00:41:57.172166 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:41:57.174610 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 273/536] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Aug 28 00:41:56 2022] GET /identity/v3/projects/service => generated 88 bytes in 242 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:41:57.175581 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-330c165f-0565-4d22-90f5-ebc98b8382cb None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:57.222117 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03f848ce-12ed-4e00-90ac-5ec94e8fbf23 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:57.271010 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f453703-bdfb-46dc-820b-d130be348c99 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:57.272492 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f453703-bdfb-46dc-820b-d130be348c99 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:57.272927 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f453703-bdfb-46dc-820b-d130be348c99 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:57.273370 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f453703-bdfb-46dc-820b-d130be348c99 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:57.274726 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f453703-bdfb-46dc-820b-d130be348c99 None admin] RBAC: Adding query filter params (role.id=348070fea3dd4cc2a6e00e29fe82cc25, scope.project.id=f53e6fdb369b480fbebb4086a505df67, user.id=168b07df50c24593ba73dbde5b037aed) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:57.275189 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f453703-bdfb-46dc-820b-d130be348c99 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:57.277722 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f453703-bdfb-46dc-820b-d130be348c99 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:57.291907 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 264/537] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Aug 28 00:41:57 2022] GET /identity/v3/role_assignments?user.id=168b07df50c24593ba73dbde5b037aed&scope.project.id=f53e6fdb369b480fbebb4086a505df67&role.id=348070fea3dd4cc2a6e00e29fe82cc25 => generated 259 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:57.291907 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f453703-bdfb-46dc-820b-d130be348c99 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:57.313607 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-330c165f-0565-4d22-90f5-ebc98b8382cb None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:57.353511 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-942f81c8-479e-455b-af17-d3943d6d1a03 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:57.354174 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-942f81c8-479e-455b-af17-d3943d6d1a03 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:57.354674 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-942f81c8-479e-455b-af17-d3943d6d1a03 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:57.354961 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-942f81c8-479e-455b-af17-d3943d6d1a03 None admin] PATH_INFO: `/v3/projects` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:57.355748 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-942f81c8-479e-455b-af17-d3943d6d1a03 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:57.356062 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-942f81c8-479e-455b-af17-d3943d6d1a03 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:57.357972 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-942f81c8-479e-455b-af17-d3943d6d1a03 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:57.373298 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 274/538] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Aug 28 00:41:57 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:57.379132 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-942f81c8-479e-455b-af17-d3943d6d1a03 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:57.473182 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:57.491835 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f453703-bdfb-46dc-820b-d130be348c99 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:57.499297 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:57.531240 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69548d34-5915-43a1-a9bd-ee8384c50623 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:57.532213 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69548d34-5915-43a1-a9bd-ee8384c50623 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:57.532676 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69548d34-5915-43a1-a9bd-ee8384c50623 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:57.533111 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69548d34-5915-43a1-a9bd-ee8384c50623 None admin] PATH_INFO: `/v3/projects` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:57.533968 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69548d34-5915-43a1-a9bd-ee8384c50623 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:57.534411 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69548d34-5915-43a1-a9bd-ee8384c50623 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:57.536120 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69548d34-5915-43a1-a9bd-ee8384c50623 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:57.560991 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-942f81c8-479e-455b-af17-d3943d6d1a03 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:57.561287 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 265/539] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Aug 28 00:41:57 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 267 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:57.563515 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69548d34-5915-43a1-a9bd-ee8384c50623 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:57.610050 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c96bee58-8830-4a1f-b6c4-edea68c2dd86 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:57.610670 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c96bee58-8830-4a1f-b6c4-edea68c2dd86 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:57.610944 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c96bee58-8830-4a1f-b6c4-edea68c2dd86 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:57.611211 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c96bee58-8830-4a1f-b6c4-edea68c2dd86 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:57.611944 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c96bee58-8830-4a1f-b6c4-edea68c2dd86 None admin] RBAC: Adding query filter params (role.id=348070fea3dd4cc2a6e00e29fe82cc25, scope.project.id=f53e6fdb369b480fbebb4086a505df67, user.id=364a28c108e744728917ba2b0aa2347d) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:57.612245 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c96bee58-8830-4a1f-b6c4-edea68c2dd86 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:57.615165 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c96bee58-8830-4a1f-b6c4-edea68c2dd86 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:57.627461 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 275/540] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Aug 28 00:41:57 2022] GET /identity/v3/role_assignments?user.id=364a28c108e744728917ba2b0aa2347d&scope.project.id=f53e6fdb369b480fbebb4086a505df67&role.id=348070fea3dd4cc2a6e00e29fe82cc25 => generated 259 bytes in 247 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:57.741245 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69548d34-5915-43a1-a9bd-ee8384c50623 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:57.801438 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-390419ea-05ac-44e5-9e8e-d4cd88c42aad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:57.802042 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-390419ea-05ac-44e5-9e8e-d4cd88c42aad None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:57.802320 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-390419ea-05ac-44e5-9e8e-d4cd88c42aad None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:57.802590 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-390419ea-05ac-44e5-9e8e-d4cd88c42aad None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:57.803386 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-390419ea-05ac-44e5-9e8e-d4cd88c42aad None admin] RBAC: Adding query filter params (role.id=348070fea3dd4cc2a6e00e29fe82cc25, scope.project.id=f53e6fdb369b480fbebb4086a505df67, user.id=879afc99ebb9402b94fadc6091040c5c) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:57.803692 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-390419ea-05ac-44e5-9e8e-d4cd88c42aad None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:57.805204 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-390419ea-05ac-44e5-9e8e-d4cd88c42aad None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:57.817847 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 266/541] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Aug 28 00:41:57 2022] GET /identity/v3/role_assignments?user.id=879afc99ebb9402b94fadc6091040c5c&scope.project.id=f53e6fdb369b480fbebb4086a505df67&role.id=348070fea3dd4cc2a6e00e29fe82cc25 => generated 259 bytes in 255 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 28 00:41:57.927033 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71868149-2d4d-48d7-b76c-3e0fa88aae3e None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:57.927558 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71868149-2d4d-48d7-b76c-3e0fa88aae3e None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:57.927558 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71868149-2d4d-48d7-b76c-3e0fa88aae3e None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:57.929568 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 276/542] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:41:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:41:57.935458 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb8198d0-7165-4039-9947-3fd30cf3ac9f None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:57.935741 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb8198d0-7165-4039-9947-3fd30cf3ac9f None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:57.936014 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb8198d0-7165-4039-9947-3fd30cf3ac9f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:57.956254 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-bb8198d0-7165-4039-9947-3fd30cf3ac9f None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:41:57.957540 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-bb8198d0-7165-4039-9947-3fd30cf3ac9f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:41:57.960939 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:58.063121 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 267/543] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:41:57 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 28 00:41:58.362001 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71868149-2d4d-48d7-b76c-3e0fa88aae3e None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:58.471593 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71868149-2d4d-48d7-b76c-3e0fa88aae3e None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:58.491334 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:58.509458 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1bbc3f4-dd0b-4fa2-8c77-1c697db892bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:58.510334 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1bbc3f4-dd0b-4fa2-8c77-1c697db892bc None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:58.510607 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1bbc3f4-dd0b-4fa2-8c77-1c697db892bc None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:58.510873 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1bbc3f4-dd0b-4fa2-8c77-1c697db892bc None admin] PATH_INFO: `/v3/roles/service` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:58.521715 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1bbc3f4-dd0b-4fa2-8c77-1c697db892bc None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 28 00:41:58.526863 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-a1bbc3f4-dd0b-4fa2-8c77-1c697db892bc None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 28 00:41:58.526863 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:41:58.526863 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:41:58.526863 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:41:58.526863 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:41:58.526863 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:41:58.526863 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:41:58.526863 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:41:58.526863 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:41:58.526863 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:41:58.526863 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:41:58.526863 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:41:58.526863 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 28 00:41:58.526863 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 28 00:41:58.526863 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 28 00:41:58.526863 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 28 00:41:58.527764 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 28 00:41:58.527764 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 28 00:41:58.527764 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 28 00:41:58.527764 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 28 00:41:58.527764 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:41:58.527764 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:41:58.527764 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:41:58.527764 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:41:58.527764 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:41:58.527764 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:41:58.527764 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:41:58.527764 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:41:58.527764 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:41:58.527764 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:41:58.527764 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:41:58.527764 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:41:58.528949 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:41:58.528949 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:41:58.528949 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:41:58.528949 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:41:58.528949 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 28 00:41:58.528949 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 28 00:41:58.528949 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 28 00:41:58.528949 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 28 00:41:58.528949 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 28 00:41:58.528949 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 28 00:41:58.528949 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 28 00:41:58.528949 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:41:58.528949 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 277/544] 10.222.0.42 () {56 vars in 1102 bytes} [Sun Aug 28 00:41:58 2022] GET /identity/v3/roles/service => generated 85 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:41:58.534347 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb8198d0-7165-4039-9947-3fd30cf3ac9f None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:58.639537 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb8198d0-7165-4039-9947-3fd30cf3ac9f None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:58.692647 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97433f79-2ef6-46b1-a540-90f5cea46624 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:58.693295 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97433f79-2ef6-46b1-a540-90f5cea46624 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:58.693571 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97433f79-2ef6-46b1-a540-90f5cea46624 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:58.693834 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97433f79-2ef6-46b1-a540-90f5cea46624 None admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:58.694492 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97433f79-2ef6-46b1-a540-90f5cea46624 None admin] RBAC: Adding query filter params (name=service) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:58.694785 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97433f79-2ef6-46b1-a540-90f5cea46624 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:58.696156 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97433f79-2ef6-46b1-a540-90f5cea46624 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:58.708059 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 268/545] 10.222.0.42 () {56 vars in 1103 bytes} [Sun Aug 28 00:41:58 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:58.713761 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1bbc3f4-dd0b-4fa2-8c77-1c697db892bc None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:58.775575 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-281bdeba-4387-46bc-b85a-a06fa1d6211a None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:58.776093 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-281bdeba-4387-46bc-b85a-a06fa1d6211a None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:58.776093 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-281bdeba-4387-46bc-b85a-a06fa1d6211a None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:58.777897 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 269/546] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:41:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:41:58.783447 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59883331-478f-4587-8da9-b8cb18456198 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:58.783750 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59883331-478f-4587-8da9-b8cb18456198 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:58.784011 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59883331-478f-4587-8da9-b8cb18456198 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:58.790794 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:58.808002 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-59883331-478f-4587-8da9-b8cb18456198 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:41:58.809532 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-59883331-478f-4587-8da9-b8cb18456198 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:41:58.914228 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1bbc3f4-dd0b-4fa2-8c77-1c697db892bc None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:58.946074 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 270/547] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:41:58 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 28 00:41:58.997204 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36a016e9-66e5-4a42-8b73-cfd49ceceab7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:58.998579 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36a016e9-66e5-4a42-8b73-cfd49ceceab7 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:58.999084 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36a016e9-66e5-4a42-8b73-cfd49ceceab7 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:58.999616 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36a016e9-66e5-4a42-8b73-cfd49ceceab7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:59.017287 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36a016e9-66e5-4a42-8b73-cfd49ceceab7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:59.018824 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36a016e9-66e5-4a42-8b73-cfd49ceceab7 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:59.038609 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-36a016e9-66e5-4a42-8b73-cfd49ceceab7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:41:59.038609 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:41:59.038609 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:41:59.038609 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:41:59.038609 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:41:59.038609 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:41:59.038609 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:41:59.038609 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:41:59.038609 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:41:59.038609 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:41:59.038609 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:41:59.038609 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:41:59.038609 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 28 00:41:59.038609 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 28 00:41:59.038609 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 28 00:41:59.038609 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 28 00:41:59.039588 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:41:59.039588 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:41:59.039588 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:41:59.039588 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:41:59.039588 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:41:59.039588 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:41:59.039588 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:41:59.039588 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:41:59.039588 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:41:59.039588 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:41:59.039588 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:41:59.039588 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:41:59.039588 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:41:59.039588 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:41:59.039588 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:41:59.039588 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:41:59.039588 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 28 00:41:59.040414 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 28 00:41:59.040414 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:41:59.040414 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:41:59.040414 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 278/548] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:41:58 2022] GET /identity/v3/domains/Default => generated 87 bytes in 327 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:41:59.047252 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59883331-478f-4587-8da9-b8cb18456198 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:59.205889 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59883331-478f-4587-8da9-b8cb18456198 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:59.250095 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:59.259127 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09c68561-eb3e-4f39-a7e3-82eebcdf4efd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:59.259803 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c68561-eb3e-4f39-a7e3-82eebcdf4efd None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:59.260245 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c68561-eb3e-4f39-a7e3-82eebcdf4efd None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:59.260723 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c68561-eb3e-4f39-a7e3-82eebcdf4efd None admin] PATH_INFO: `/v3/domains` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:59.261402 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09c68561-eb3e-4f39-a7e3-82eebcdf4efd None admin] RBAC: Adding query filter params (name=Default) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:59.261707 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09c68561-eb3e-4f39-a7e3-82eebcdf4efd None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:59.263110 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09c68561-eb3e-4f39-a7e3-82eebcdf4efd None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:59.279664 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 271/549] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Aug 28 00:41:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 233 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:59.285257 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36a016e9-66e5-4a42-8b73-cfd49ceceab7 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:59.293545 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09c68561-eb3e-4f39-a7e3-82eebcdf4efd None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:59.366297 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:59.427120 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09c68561-eb3e-4f39-a7e3-82eebcdf4efd None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:59.489450 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71c35f62-1c61-4066-ac81-9faa12b2252e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:59.490014 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71c35f62-1c61-4066-ac81-9faa12b2252e None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:59.490293 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71c35f62-1c61-4066-ac81-9faa12b2252e None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:59.490566 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71c35f62-1c61-4066-ac81-9faa12b2252e None admin] PATH_INFO: `/v3/roles/service` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:59.492855 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36a016e9-66e5-4a42-8b73-cfd49ceceab7 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:59.514420 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71c35f62-1c61-4066-ac81-9faa12b2252e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 28 00:41:59.521682 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-71c35f62-1c61-4066-ac81-9faa12b2252e None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 28 00:41:59.521682 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:41:59.521682 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:41:59.521682 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:41:59.521682 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:41:59.521682 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:41:59.521682 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:41:59.521682 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:41:59.521682 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:41:59.521682 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:41:59.521682 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:41:59.521682 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:41:59.521682 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 28 00:41:59.521682 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 28 00:41:59.521682 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 28 00:41:59.521682 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 28 00:41:59.523357 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 28 00:41:59.523357 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 28 00:41:59.523357 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 28 00:41:59.523357 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 28 00:41:59.523357 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:41:59.523357 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:41:59.523357 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:41:59.523357 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:41:59.523357 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:41:59.523357 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:41:59.523357 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:41:59.523357 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:41:59.523357 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:41:59.523357 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:41:59.523357 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:41:59.523357 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:41:59.524323 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:41:59.524323 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:41:59.524323 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:41:59.524323 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:41:59.524323 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 28 00:41:59.524323 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 28 00:41:59.524323 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 28 00:41:59.524323 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 28 00:41:59.524323 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 28 00:41:59.524323 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 28 00:41:59.524323 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 28 00:41:59.524323 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:41:59.533047 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 272/550] 10.222.0.42 () {56 vars in 1102 bytes} [Sun Aug 28 00:41:59 2022] GET /identity/v3/roles/service => generated 85 bytes in 230 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 28 00:41:59.534037 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71c35f62-1c61-4066-ac81-9faa12b2252e None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:59.569677 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89e8497c-60e6-4052-a050-7bad3e6bb42d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:59.570321 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89e8497c-60e6-4052-a050-7bad3e6bb42d None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:59.570597 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89e8497c-60e6-4052-a050-7bad3e6bb42d None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:59.570868 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89e8497c-60e6-4052-a050-7bad3e6bb42d None admin] PATH_INFO: `/v3/users/swift` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:59.587899 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89e8497c-60e6-4052-a050-7bad3e6bb42d None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:59.593013 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89e8497c-60e6-4052-a050-7bad3e6bb42d None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:59.604551 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-89e8497c-60e6-4052-a050-7bad3e6bb42d None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Aug 28 00:41:59.604551 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:41:59.604551 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:41:59.604551 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:41:59.604551 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:41:59.604551 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:41:59.604551 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:41:59.604551 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:41:59.604551 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:41:59.604551 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:41:59.604551 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:41:59.604551 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:41:59.604551 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 28 00:41:59.604551 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 28 00:41:59.604551 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 28 00:41:59.604551 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 28 00:41:59.605755 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:41:59.605755 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:41:59.605755 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 28 00:41:59.605755 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 28 00:41:59.605755 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 28 00:41:59.605755 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 28 00:41:59.605755 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:41:59.605755 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:41:59.605755 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:41:59.605755 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:41:59.605755 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:41:59.605755 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:41:59.605755 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:41:59.605755 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:41:59.605755 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:41:59.605755 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:41:59.605755 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:41:59.605755 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:41:59.607046 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:41:59.607046 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:41:59.607046 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 28 00:41:59.607046 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 28 00:41:59.607046 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 28 00:41:59.607046 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 28 00:41:59.607046 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 28 00:41:59.607046 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 28 00:41:59.607046 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Aug 28 00:41:59.607046 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:41:59.609464 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 279/551] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Aug 28 00:41:59 2022] GET /identity/v3/users/swift => generated 83 bytes in 325 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:41:59.615546 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89e8497c-60e6-4052-a050-7bad3e6bb42d None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:59.645339 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:59.724305 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:59.731110 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89e8497c-60e6-4052-a050-7bad3e6bb42d None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:59.770696 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ec66d62-6e42-4bb4-892a-3db42d55f782 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:59.771591 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec66d62-6e42-4bb4-892a-3db42d55f782 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:59.772094 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec66d62-6e42-4bb4-892a-3db42d55f782 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:59.772535 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec66d62-6e42-4bb4-892a-3db42d55f782 None admin] PATH_INFO: `/v3/users` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:59.773511 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ec66d62-6e42-4bb4-892a-3db42d55f782 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:59.774129 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ec66d62-6e42-4bb4-892a-3db42d55f782 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:59.775835 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ec66d62-6e42-4bb4-892a-3db42d55f782 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:59.794934 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 280/552] 10.222.0.42 () {56 vars in 1135 bytes} [Sun Aug 28 00:41:59 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 28 00:41:59.801477 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ec66d62-6e42-4bb4-892a-3db42d55f782 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:41:59.866990 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71c35f62-1c61-4066-ac81-9faa12b2252e None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:59.908985 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ec66d62-6e42-4bb4-892a-3db42d55f782 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:41:59.949627 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c30cbb2-2f52-4c80-9190-acb698c9ed5d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:59.950538 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c30cbb2-2f52-4c80-9190-acb698c9ed5d None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:59.951041 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c30cbb2-2f52-4c80-9190-acb698c9ed5d None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:59.951512 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c30cbb2-2f52-4c80-9190-acb698c9ed5d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:59.960265 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b5cc4d3-5816-4b25-a106-0a24b93ffe12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:41:59.961025 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b5cc4d3-5816-4b25-a106-0a24b93ffe12 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:59.961486 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b5cc4d3-5816-4b25-a106-0a24b93ffe12 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:59.961990 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b5cc4d3-5816-4b25-a106-0a24b93ffe12 None admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:59.962893 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b5cc4d3-5816-4b25-a106-0a24b93ffe12 None admin] RBAC: Adding query filter params (name=service) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:41:59.963450 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b5cc4d3-5816-4b25-a106-0a24b93ffe12 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:41:59.965320 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b5cc4d3-5816-4b25-a106-0a24b93ffe12 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:41:59.972596 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:41:59.980770 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 273/553] 10.222.0.42 () {56 vars in 1103 bytes} [Sun Aug 28 00:41:59 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 444 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:41:59.980770 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3c633ce-9ed2-47ec-9882-69db25d77fca None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:41:59.980770 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3c633ce-9ed2-47ec-9882-69db25d77fca None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:41:59.980770 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3c633ce-9ed2-47ec-9882-69db25d77fca None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:41:59.980770 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 274/554] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:41:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:41:59.986278 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3c633ce-9ed2-47ec-9882-69db25d77fca None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:00.039538 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c30cbb2-2f52-4c80-9190-acb698c9ed5d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:00.041828 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c30cbb2-2f52-4c80-9190-acb698c9ed5d None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:00.093905 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3c633ce-9ed2-47ec-9882-69db25d77fca None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:00.107804 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:00.133573 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65192149-6599-439c-bacc-710c4d2e0fcb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:00.134210 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65192149-6599-439c-bacc-710c4d2e0fcb None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:00.134492 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65192149-6599-439c-bacc-710c4d2e0fcb None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:00.134762 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65192149-6599-439c-bacc-710c4d2e0fcb None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:00.145254 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65192149-6599-439c-bacc-710c4d2e0fcb None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:00.146709 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65192149-6599-439c-bacc-710c4d2e0fcb None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:00.151865 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 281/555] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Aug 28 00:41:59 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 346 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Aug 28 00:42:00.151865 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0b6273a-e1d4-4505-9600-4f1d6414f507 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:00.151865 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0b6273a-e1d4-4505-9600-4f1d6414f507 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:00.151865 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0b6273a-e1d4-4505-9600-4f1d6414f507 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:00.161620 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-65192149-6599-439c-bacc-710c4d2e0fcb None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:42:00.161620 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:00.161620 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:00.161620 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:00.161620 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:00.161620 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:00.161620 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:00.161620 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:00.161620 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:00.161620 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:00.161620 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:00.161620 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:00.161620 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 28 00:42:00.161620 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 28 00:42:00.161620 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 28 00:42:00.161620 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 28 00:42:00.162670 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:00.162670 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:00.162670 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:00.162670 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:00.162670 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:00.162670 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:00.162670 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:00.162670 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:00.162670 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:00.162670 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:00.162670 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:00.162670 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:00.162670 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:00.162670 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:00.162670 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:00.162670 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:00.162670 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 28 00:42:00.163550 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 28 00:42:00.163550 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:42:00.163550 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:42:00.163550 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 275/556] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:41:59 2022] GET /identity/v3/domains/Default => generated 87 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:00.165251 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3ca8d2e-d364-4937-94c5-bef6fab6e25c None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:00.165540 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3ca8d2e-d364-4937-94c5-bef6fab6e25c None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:00.165817 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3ca8d2e-d364-4937-94c5-bef6fab6e25c None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:00.167249 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-d0b6273a-e1d4-4505-9600-4f1d6414f507 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:42:00.167373 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 276/557] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:42:00 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:42:00.168406 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e8d89dd-da1f-4e4c-858e-b4c8da705f9e None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:00.168508 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-d0b6273a-e1d4-4505-9600-4f1d6414f507 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:42:00.168692 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e8d89dd-da1f-4e4c-858e-b4c8da705f9e None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:00.168951 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e8d89dd-da1f-4e4c-858e-b4c8da705f9e None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:00.170326 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 277/558] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:42:00 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:42:00.170884 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e8d89dd-da1f-4e4c-858e-b4c8da705f9e None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:00.217703 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:00.249337 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 282/559] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:42:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 28 00:42:00.250290 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0b6273a-e1d4-4505-9600-4f1d6414f507 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:00.283968 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e8d89dd-da1f-4e4c-858e-b4c8da705f9e None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:00.321870 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7da47cb7-a0fd-468d-ba8f-6af78c5d4577 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:00.322439 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7da47cb7-a0fd-468d-ba8f-6af78c5d4577 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:00.322943 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7da47cb7-a0fd-468d-ba8f-6af78c5d4577 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:00.322943 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7da47cb7-a0fd-468d-ba8f-6af78c5d4577 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:00.336081 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7da47cb7-a0fd-468d-ba8f-6af78c5d4577 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:00.337723 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7da47cb7-a0fd-468d-ba8f-6af78c5d4577 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:00.352526 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-7da47cb7-a0fd-468d-ba8f-6af78c5d4577 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:42:00.352526 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:00.352526 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:00.352526 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:00.352526 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:00.352526 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:00.352526 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:00.352526 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:00.352526 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:00.352526 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:00.352526 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:00.352526 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:00.352526 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 28 00:42:00.352526 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 28 00:42:00.352526 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 28 00:42:00.352526 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 28 00:42:00.353550 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:00.353550 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:00.353550 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:00.353550 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:00.353550 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:00.353550 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:00.353550 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:00.353550 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:00.353550 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:00.353550 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:00.353550 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:00.353550 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:00.353550 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:00.353550 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:00.353550 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:00.353550 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:00.353550 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 28 00:42:00.354459 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 28 00:42:00.354459 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:42:00.354459 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:42:00.354459 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 278/560] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:42:00 2022] GET /identity/v3/domains/Default => generated 87 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:00.355299 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bdf7055-cd8d-4a5c-a8c7-eb03ab01f3aa None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:00.355548 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bdf7055-cd8d-4a5c-a8c7-eb03ab01f3aa None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:00.355822 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bdf7055-cd8d-4a5c-a8c7-eb03ab01f3aa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:00.364175 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0b6273a-e1d4-4505-9600-4f1d6414f507 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:00.383840 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-9bdf7055-cd8d-4a5c-a8c7-eb03ab01f3aa None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:42:00.385083 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-9bdf7055-cd8d-4a5c-a8c7-eb03ab01f3aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:42:00.399585 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:00.403958 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e8effbf-15f5-4f5a-a7b3-3b99287a4daa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:00.404595 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e8effbf-15f5-4f5a-a7b3-3b99287a4daa None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:00.404871 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e8effbf-15f5-4f5a-a7b3-3b99287a4daa None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:00.405174 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e8effbf-15f5-4f5a-a7b3-3b99287a4daa None admin] PATH_INFO: `/v3/domains` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:00.405849 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e8effbf-15f5-4f5a-a7b3-3b99287a4daa None admin] RBAC: Adding query filter params (name=Default) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:00.406140 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e8effbf-15f5-4f5a-a7b3-3b99287a4daa None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:00.407542 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e8effbf-15f5-4f5a-a7b3-3b99287a4daa None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:00.420649 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 283/561] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Aug 28 00:42:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:00.420649 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad08a91b-fb1b-4d96-94ce-2d8e9d9cb745 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:00.420649 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad08a91b-fb1b-4d96-94ce-2d8e9d9cb745 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:00.420649 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad08a91b-fb1b-4d96-94ce-2d8e9d9cb745 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:00.446877 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-ad08a91b-fb1b-4d96-94ce-2d8e9d9cb745 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:42:00.448277 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-ad08a91b-fb1b-4d96-94ce-2d8e9d9cb745 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:42:00.464813 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 279/562] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:42:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 28 00:42:00.465146 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bdf7055-cd8d-4a5c-a8c7-eb03ab01f3aa None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:00.496759 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:00.542833 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 284/563] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:42:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 28 00:42:00.544013 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad08a91b-fb1b-4d96-94ce-2d8e9d9cb745 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:00.589038 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bdf7055-cd8d-4a5c-a8c7-eb03ab01f3aa None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:00.625903 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e806b824-6cf1-414c-9f11-986a83a6fa76 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:00.626906 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e806b824-6cf1-414c-9f11-986a83a6fa76 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:00.627356 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e806b824-6cf1-414c-9f11-986a83a6fa76 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:00.627814 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e806b824-6cf1-414c-9f11-986a83a6fa76 None admin] PATH_INFO: `/v3/domains` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:00.628741 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e806b824-6cf1-414c-9f11-986a83a6fa76 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:00.629249 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e806b824-6cf1-414c-9f11-986a83a6fa76 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:00.631105 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e806b824-6cf1-414c-9f11-986a83a6fa76 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:00.643615 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 280/564] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Aug 28 00:42:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:00.644998 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e806b824-6cf1-414c-9f11-986a83a6fa76 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:00.663482 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:00.666646 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad08a91b-fb1b-4d96-94ce-2d8e9d9cb745 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:00.712427 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d16199e7-0bc5-4cc8-9e33-7140a63ca68b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:00.713349 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d16199e7-0bc5-4cc8-9e33-7140a63ca68b None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:00.713532 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d16199e7-0bc5-4cc8-9e33-7140a63ca68b None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:00.713807 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d16199e7-0bc5-4cc8-9e33-7140a63ca68b None admin] PATH_INFO: `/v3/users/glance` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:00.721700 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d16199e7-0bc5-4cc8-9e33-7140a63ca68b None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:00.723328 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d16199e7-0bc5-4cc8-9e33-7140a63ca68b None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:00.736332 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-d16199e7-0bc5-4cc8-9e33-7140a63ca68b None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Aug 28 00:42:00.736332 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:00.736332 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:00.736332 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:00.736332 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:00.736332 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:00.736332 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:00.736332 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:00.736332 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:00.736332 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:00.736332 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:00.736332 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:00.736332 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 28 00:42:00.736332 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 28 00:42:00.736332 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 28 00:42:00.736332 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 28 00:42:00.737355 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:00.737355 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:00.737355 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 28 00:42:00.737355 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 28 00:42:00.737355 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 28 00:42:00.737355 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 28 00:42:00.737355 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:00.737355 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:00.737355 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:00.737355 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:00.737355 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:00.737355 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:00.737355 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:00.737355 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:00.737355 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:00.737355 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:00.737355 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:00.737355 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:00.738635 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:00.738635 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:00.738635 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 28 00:42:00.738635 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 28 00:42:00.738635 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 28 00:42:00.738635 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 28 00:42:00.738635 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 28 00:42:00.738635 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 28 00:42:00.738635 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Aug 28 00:42:00.738635 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:00.738635 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 285/565] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Aug 28 00:42:00 2022] GET /identity/v3/users/glance => generated 84 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:00.738635 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d16199e7-0bc5-4cc8-9e33-7140a63ca68b None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:00.769893 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:00.776724 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e806b824-6cf1-414c-9f11-986a83a6fa76 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:00.821577 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c07fad2-43a2-4d48-a583-afb095773a5e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:00.822284 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c07fad2-43a2-4d48-a583-afb095773a5e None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:00.822558 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c07fad2-43a2-4d48-a583-afb095773a5e None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:00.822825 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c07fad2-43a2-4d48-a583-afb095773a5e None admin] PATH_INFO: `/v3/roles/service` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:00.835910 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c07fad2-43a2-4d48-a583-afb095773a5e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 28 00:42:00.841432 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-9c07fad2-43a2-4d48-a583-afb095773a5e None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 28 00:42:00.841432 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:00.841432 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:00.841432 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:00.841432 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:00.841432 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:00.841432 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:00.841432 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:00.841432 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:00.841432 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:00.841432 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:00.841432 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:00.841432 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 28 00:42:00.841432 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 28 00:42:00.841432 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 28 00:42:00.841432 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 28 00:42:00.842589 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 28 00:42:00.842589 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 28 00:42:00.842589 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 28 00:42:00.842589 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 28 00:42:00.842589 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:00.842589 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:00.842589 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:00.842589 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:00.842589 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:00.842589 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:00.842589 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:00.842589 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:00.842589 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:00.842589 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:00.842589 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:00.842589 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:00.843546 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:00.843546 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:00.843546 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:00.843546 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:00.843546 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 28 00:42:00.843546 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 28 00:42:00.843546 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 28 00:42:00.843546 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 28 00:42:00.843546 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 28 00:42:00.843546 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 28 00:42:00.843546 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 28 00:42:00.843546 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:42:00.843546 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 281/566] 10.222.0.42 () {56 vars in 1102 bytes} [Sun Aug 28 00:42:00 2022] GET /identity/v3/roles/service => generated 85 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:00.844266 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c07fad2-43a2-4d48-a583-afb095773a5e None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:00.870958 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d16199e7-0bc5-4cc8-9e33-7140a63ca68b None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:00.904361 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-686a40dc-805a-4eae-8d97-e300e9786020 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:00.905010 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-686a40dc-805a-4eae-8d97-e300e9786020 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:00.905291 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-686a40dc-805a-4eae-8d97-e300e9786020 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:00.905625 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-686a40dc-805a-4eae-8d97-e300e9786020 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:00.909325 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:00.912587 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-686a40dc-805a-4eae-8d97-e300e9786020 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:00.914101 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-686a40dc-805a-4eae-8d97-e300e9786020 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:00.925305 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-686a40dc-805a-4eae-8d97-e300e9786020 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 28 00:42:00.925305 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:00.925305 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:00.925305 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:00.925305 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:00.925305 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:00.925305 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:00.925305 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:00.925305 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:00.925305 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:00.925305 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:00.925305 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:00.925305 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 28 00:42:00.925305 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 28 00:42:00.925305 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 28 00:42:00.926370 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 28 00:42:00.926370 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:00.926370 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:00.926370 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:00.926370 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:00.926370 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:00.926370 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:00.926370 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:00.926370 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:00.926370 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:00.926370 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:00.926370 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:00.926370 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:00.926370 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:00.926370 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:00.926370 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:00.926370 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:00.927249 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 28 00:42:00.927249 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 28 00:42:00.927249 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 28 00:42:00.927249 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 28 00:42:00.927249 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 28 00:42:00.927249 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 28 00:42:00.927249 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 28 00:42:00.927249 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:00.927249 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 286/567] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Aug 28 00:42:00 2022] GET /identity/v3/projects/service => generated 88 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:00.927249 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-686a40dc-805a-4eae-8d97-e300e9786020 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:00.947111 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c07fad2-43a2-4d48-a583-afb095773a5e None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:00.979981 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-891b480c-580a-4966-9292-897336ecadbb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:00.980613 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-891b480c-580a-4966-9292-897336ecadbb None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:00.980919 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-891b480c-580a-4966-9292-897336ecadbb None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:00.981220 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-891b480c-580a-4966-9292-897336ecadbb None admin] PATH_INFO: `/v3/roles/service` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:00.991426 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-891b480c-580a-4966-9292-897336ecadbb None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 28 00:42:00.996684 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-891b480c-580a-4966-9292-897336ecadbb None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 28 00:42:00.996684 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:00.996684 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:00.996684 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:00.996684 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:00.996684 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:00.996684 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:00.996684 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:00.996684 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:00.996684 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:00.996684 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:00.996684 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:00.996684 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 28 00:42:00.996684 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 28 00:42:00.996684 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 28 00:42:00.996684 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 28 00:42:00.997632 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 28 00:42:00.997632 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 28 00:42:00.997632 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 28 00:42:00.997632 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 28 00:42:00.997632 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:00.997632 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:00.997632 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:00.997632 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:00.997632 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:00.997632 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:00.997632 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:00.997632 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:00.997632 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:00.997632 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:00.997632 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:00.997632 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:00.998447 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:00.998447 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:00.998447 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:00.998447 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:00.998447 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 28 00:42:00.998447 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 28 00:42:00.998447 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 28 00:42:00.998447 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 28 00:42:00.998447 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 28 00:42:00.998447 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 28 00:42:00.998447 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 28 00:42:00.998447 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:42:00.998447 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 282/568] 10.222.0.42 () {56 vars in 1102 bytes} [Sun Aug 28 00:42:00 2022] GET /identity/v3/roles/service => generated 85 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:00.999159 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-891b480c-580a-4966-9292-897336ecadbb None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:01.014931 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:01.027893 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-686a40dc-805a-4eae-8d97-e300e9786020 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:01.064821 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d8f67a3-d14b-4498-9f40-643da90a72ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:01.065538 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d8f67a3-d14b-4498-9f40-643da90a72ce None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:01.065718 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d8f67a3-d14b-4498-9f40-643da90a72ce None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:01.065986 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d8f67a3-d14b-4498-9f40-643da90a72ce None admin] PATH_INFO: `/v3/users` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:01.066709 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d8f67a3-d14b-4498-9f40-643da90a72ce None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:01.067009 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d8f67a3-d14b-4498-9f40-643da90a72ce None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:01.068420 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d8f67a3-d14b-4498-9f40-643da90a72ce None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:01.085461 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 287/569] 10.222.0.42 () {56 vars in 1137 bytes} [Sun Aug 28 00:42:00 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:01.086275 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d8f67a3-d14b-4498-9f40-643da90a72ce None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:01.102266 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-891b480c-580a-4966-9292-897336ecadbb None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:01.135983 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:01.145499 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54c0d5df-846d-48c3-a7fb-a000a357548b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:01.146132 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c0d5df-846d-48c3-a7fb-a000a357548b None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:01.146410 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c0d5df-846d-48c3-a7fb-a000a357548b None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:01.146675 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c0d5df-846d-48c3-a7fb-a000a357548b None admin] PATH_INFO: `/v3/roles/service` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:01.157046 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54c0d5df-846d-48c3-a7fb-a000a357548b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 28 00:42:01.162652 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-54c0d5df-846d-48c3-a7fb-a000a357548b None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 28 00:42:01.162652 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:01.162652 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:01.162652 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:01.162652 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:01.162652 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:01.162652 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:01.162652 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:01.162652 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:01.162652 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:01.162652 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:01.162652 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:01.162652 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 28 00:42:01.162652 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 28 00:42:01.162652 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 28 00:42:01.162652 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 28 00:42:01.163618 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 28 00:42:01.163618 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 28 00:42:01.163618 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 28 00:42:01.163618 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 28 00:42:01.163618 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:01.163618 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:01.163618 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:01.163618 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:01.163618 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:01.163618 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:01.163618 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:01.163618 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:01.163618 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:01.163618 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:01.163618 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:01.163618 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:01.164609 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:01.164609 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:01.164609 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:01.164609 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:01.164609 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 28 00:42:01.164609 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 28 00:42:01.164609 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 28 00:42:01.164609 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 28 00:42:01.164609 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 28 00:42:01.164609 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 28 00:42:01.164609 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 28 00:42:01.164609 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:42:01.164609 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 283/570] 10.222.0.42 () {56 vars in 1102 bytes} [Sun Aug 28 00:42:00 2022] GET /identity/v3/roles/service => generated 85 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:01.165330 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54c0d5df-846d-48c3-a7fb-a000a357548b None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:01.213823 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d8f67a3-d14b-4498-9f40-643da90a72ce None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:01.249326 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8dcc325-08b8-4faa-8ec9-599663553469 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:01.250043 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8dcc325-08b8-4faa-8ec9-599663553469 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:01.250386 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8dcc325-08b8-4faa-8ec9-599663553469 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:01.250592 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8dcc325-08b8-4faa-8ec9-599663553469 None admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:01.251282 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8dcc325-08b8-4faa-8ec9-599663553469 None admin] RBAC: Adding query filter params (name=service) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:01.251577 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8dcc325-08b8-4faa-8ec9-599663553469 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:01.253339 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8dcc325-08b8-4faa-8ec9-599663553469 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:01.262418 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 288/571] 10.222.0.42 () {56 vars in 1103 bytes} [Sun Aug 28 00:42:01 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:01.263334 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8dcc325-08b8-4faa-8ec9-599663553469 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:01.276393 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:01.296634 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54c0d5df-846d-48c3-a7fb-a000a357548b None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:01.332219 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51dcf186-e493-41cc-9542-7ecba3eff1be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:01.332827 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51dcf186-e493-41cc-9542-7ecba3eff1be None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:01.333129 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51dcf186-e493-41cc-9542-7ecba3eff1be None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:01.333400 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51dcf186-e493-41cc-9542-7ecba3eff1be None admin] PATH_INFO: `/v3/projects` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:01.334091 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51dcf186-e493-41cc-9542-7ecba3eff1be None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:01.334390 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51dcf186-e493-41cc-9542-7ecba3eff1be None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:01.335852 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51dcf186-e493-41cc-9542-7ecba3eff1be None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:01.348766 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 284/572] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Aug 28 00:42:01 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:01.350236 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51dcf186-e493-41cc-9542-7ecba3eff1be None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:01.390461 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8dcc325-08b8-4faa-8ec9-599663553469 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:01.409370 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:01.428726 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a057966-1907-4c9a-9f78-a4cea19e37fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:01.429373 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a057966-1907-4c9a-9f78-a4cea19e37fd None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:01.429654 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a057966-1907-4c9a-9f78-a4cea19e37fd None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:01.429930 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a057966-1907-4c9a-9f78-a4cea19e37fd None admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:01.430636 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a057966-1907-4c9a-9f78-a4cea19e37fd None admin] RBAC: Adding query filter params (name=service) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:01.430943 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a057966-1907-4c9a-9f78-a4cea19e37fd None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:01.432350 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a057966-1907-4c9a-9f78-a4cea19e37fd None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:01.441923 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 289/573] 10.222.0.42 () {56 vars in 1103 bytes} [Sun Aug 28 00:42:01 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:01.442811 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a057966-1907-4c9a-9f78-a4cea19e37fd None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:01.465655 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51dcf186-e493-41cc-9542-7ecba3eff1be None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:01.505854 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a415238e-f9b7-44fa-aee3-6b52b0c9fb50 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:01.506512 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a415238e-f9b7-44fa-aee3-6b52b0c9fb50 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:01.506790 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a415238e-f9b7-44fa-aee3-6b52b0c9fb50 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:01.507055 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a415238e-f9b7-44fa-aee3-6b52b0c9fb50 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:01.518289 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a415238e-f9b7-44fa-aee3-6b52b0c9fb50 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:01.519441 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:01.520050 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a415238e-f9b7-44fa-aee3-6b52b0c9fb50 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:01.536115 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-a415238e-f9b7-44fa-aee3-6b52b0c9fb50 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:42:01.536115 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:01.536115 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:01.536115 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:01.536115 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:01.536115 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:01.536115 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:01.536115 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:01.536115 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:01.536115 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:01.536115 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:01.536115 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:01.536115 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 28 00:42:01.536115 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 28 00:42:01.536115 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 28 00:42:01.536115 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 28 00:42:01.540216 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:01.540216 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:01.540216 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:01.540216 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:01.540216 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:01.540216 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:01.540216 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:01.540216 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:01.540216 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:01.540216 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:01.540216 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:01.540216 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:01.540216 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:01.540216 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:01.540216 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:01.540216 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:01.540216 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 28 00:42:01.541337 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 28 00:42:01.541337 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:42:01.541337 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:42:01.541337 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 285/574] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:42:01 2022] GET /identity/v3/domains/Default => generated 87 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:01.541337 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a415238e-f9b7-44fa-aee3-6b52b0c9fb50 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:01.559944 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a057966-1907-4c9a-9f78-a4cea19e37fd None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:01.600166 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-288ec3a0-7897-412c-a42d-6f803f04c7e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:01.600748 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-288ec3a0-7897-412c-a42d-6f803f04c7e3 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:01.601053 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-288ec3a0-7897-412c-a42d-6f803f04c7e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:01.601328 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-288ec3a0-7897-412c-a42d-6f803f04c7e3 None admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:01.601989 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-288ec3a0-7897-412c-a42d-6f803f04c7e3 None admin] RBAC: Adding query filter params (name=service) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:01.602290 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-288ec3a0-7897-412c-a42d-6f803f04c7e3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:01.603661 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-288ec3a0-7897-412c-a42d-6f803f04c7e3 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:01.611834 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 290/575] 10.222.0.42 () {56 vars in 1103 bytes} [Sun Aug 28 00:42:01 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:01.612676 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-288ec3a0-7897-412c-a42d-6f803f04c7e3 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:01.649371 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:01.652418 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a415238e-f9b7-44fa-aee3-6b52b0c9fb50 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:01.687386 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e34d1af8-018a-4c58-89f7-4ef93a43833f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:01.688022 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e34d1af8-018a-4c58-89f7-4ef93a43833f None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:01.688302 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e34d1af8-018a-4c58-89f7-4ef93a43833f None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:01.688572 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e34d1af8-018a-4c58-89f7-4ef93a43833f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:01.699339 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e34d1af8-018a-4c58-89f7-4ef93a43833f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:01.700802 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e34d1af8-018a-4c58-89f7-4ef93a43833f None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:01.715601 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-e34d1af8-018a-4c58-89f7-4ef93a43833f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:42:01.715601 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:01.715601 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:01.715601 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:01.715601 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:01.715601 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:01.715601 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:01.715601 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:01.715601 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:01.715601 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:01.715601 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:01.715601 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:01.715601 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 28 00:42:01.715601 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 28 00:42:01.715601 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 28 00:42:01.715601 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 28 00:42:01.716641 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:01.716641 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:01.716641 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:01.716641 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:01.716641 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:01.716641 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:01.716641 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:01.716641 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:01.716641 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:01.716641 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:01.716641 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:01.716641 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:01.716641 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:01.716641 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:01.716641 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:01.716641 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:01.716641 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 28 00:42:01.717573 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 28 00:42:01.717573 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:42:01.717573 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:42:01.717573 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 286/576] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:42:01 2022] GET /identity/v3/domains/Default => generated 87 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:01.717573 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e34d1af8-018a-4c58-89f7-4ef93a43833f None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:01.717876 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-288ec3a0-7897-412c-a42d-6f803f04c7e3 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:01.748828 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:01.754407 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf86a101-3228-4eca-a6ad-a63f728b5408 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:01.755015 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf86a101-3228-4eca-a6ad-a63f728b5408 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:01.755295 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf86a101-3228-4eca-a6ad-a63f728b5408 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:01.755566 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf86a101-3228-4eca-a6ad-a63f728b5408 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:01.756293 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf86a101-3228-4eca-a6ad-a63f728b5408 None admin] RBAC: Adding query filter params (role.id=348070fea3dd4cc2a6e00e29fe82cc25, scope.project.id=f53e6fdb369b480fbebb4086a505df67, user.id=ac4c5584582f4f55a7085af4cbbabf9b) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:01.756596 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf86a101-3228-4eca-a6ad-a63f728b5408 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:01.758063 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf86a101-3228-4eca-a6ad-a63f728b5408 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:01.763594 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 291/577] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Aug 28 00:42:01 2022] GET /identity/v3/role_assignments?user.id=ac4c5584582f4f55a7085af4cbbabf9b&scope.project.id=f53e6fdb369b480fbebb4086a505df67&role.id=348070fea3dd4cc2a6e00e29fe82cc25 => generated 259 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:01.764389 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf86a101-3228-4eca-a6ad-a63f728b5408 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:01.829452 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e34d1af8-018a-4c58-89f7-4ef93a43833f None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:01.869127 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39e67014-150c-45f8-87d1-49a297fbb5a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:01.869796 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39e67014-150c-45f8-87d1-49a297fbb5a7 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:01.870077 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39e67014-150c-45f8-87d1-49a297fbb5a7 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:01.870369 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39e67014-150c-45f8-87d1-49a297fbb5a7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:01.878217 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf86a101-3228-4eca-a6ad-a63f728b5408 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:01.884029 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39e67014-150c-45f8-87d1-49a297fbb5a7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:01.885666 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39e67014-150c-45f8-87d1-49a297fbb5a7 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:01.902800 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-39e67014-150c-45f8-87d1-49a297fbb5a7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:42:01.902800 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:01.902800 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:01.902800 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:01.902800 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:01.902800 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:01.902800 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:01.902800 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:01.902800 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:01.902800 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:01.902800 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:01.902800 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:01.902800 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 28 00:42:01.902800 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 28 00:42:01.902800 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 28 00:42:01.902800 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 28 00:42:01.903788 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:01.903788 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:01.903788 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:01.903788 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:01.903788 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:01.903788 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:01.903788 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:01.903788 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:01.903788 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:01.903788 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:01.903788 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:01.903788 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:01.903788 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:01.903788 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:01.903788 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:01.903788 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:01.903788 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 28 00:42:01.904668 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 28 00:42:01.904668 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:42:01.904668 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:42:01.904668 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 287/578] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:42:01 2022] GET /identity/v3/domains/Default => generated 87 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:01.904933 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39e67014-150c-45f8-87d1-49a297fbb5a7 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:01.909727 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:01.925416 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-792722bb-0ee9-4170-b383-987af7c1a4e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:01.926066 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-792722bb-0ee9-4170-b383-987af7c1a4e3 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:01.926316 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-792722bb-0ee9-4170-b383-987af7c1a4e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:01.926586 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-792722bb-0ee9-4170-b383-987af7c1a4e3 None admin] PATH_INFO: `/v3/domains` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:01.927233 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-792722bb-0ee9-4170-b383-987af7c1a4e3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:01.927560 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-792722bb-0ee9-4170-b383-987af7c1a4e3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:01.928914 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-792722bb-0ee9-4170-b383-987af7c1a4e3 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:01.942330 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 292/579] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Aug 28 00:42:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:01.942330 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-792722bb-0ee9-4170-b383-987af7c1a4e3 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:02.006751 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:02.021204 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39e67014-150c-45f8-87d1-49a297fbb5a7 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:02.053003 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-792722bb-0ee9-4170-b383-987af7c1a4e3 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:02.056364 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6af9f9f0-a8fa-4e24-8ce7-3522cf681b3e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:02.057017 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6af9f9f0-a8fa-4e24-8ce7-3522cf681b3e None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:02.057292 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6af9f9f0-a8fa-4e24-8ce7-3522cf681b3e None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:02.057557 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6af9f9f0-a8fa-4e24-8ce7-3522cf681b3e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:02.069869 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6af9f9f0-a8fa-4e24-8ce7-3522cf681b3e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:02.071474 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6af9f9f0-a8fa-4e24-8ce7-3522cf681b3e None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:02.087296 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-6af9f9f0-a8fa-4e24-8ce7-3522cf681b3e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:42:02.087296 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:02.087296 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:02.087296 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:02.087296 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:02.087296 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:02.087296 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:02.087296 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:02.087296 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:02.087296 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:02.087296 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:02.087296 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:02.087296 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 28 00:42:02.087296 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 28 00:42:02.087296 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 28 00:42:02.087296 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 28 00:42:02.088312 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:02.088312 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:02.088312 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:02.088312 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:02.088312 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:02.088312 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:02.088312 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:02.088312 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:02.088312 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:02.088312 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:02.088312 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:02.088312 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:02.088312 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:02.088312 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:02.088312 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:02.088312 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:02.088312 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 28 00:42:02.089230 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 28 00:42:02.089230 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:42:02.089230 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:42:02.089230 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 288/580] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:42:01 2022] GET /identity/v3/domains/Default => generated 87 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:02.089230 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6af9f9f0-a8fa-4e24-8ce7-3522cf681b3e None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:02.090443 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3f756be-04f1-46c8-bd35-2ad9f87d535c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:02.091052 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3f756be-04f1-46c8-bd35-2ad9f87d535c None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:02.091325 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3f756be-04f1-46c8-bd35-2ad9f87d535c None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:02.091588 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3f756be-04f1-46c8-bd35-2ad9f87d535c None admin] PATH_INFO: `/v3/domains` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:02.092227 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3f756be-04f1-46c8-bd35-2ad9f87d535c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:02.092525 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3f756be-04f1-46c8-bd35-2ad9f87d535c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:02.094001 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3f756be-04f1-46c8-bd35-2ad9f87d535c None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:02.111247 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 293/581] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Aug 28 00:42:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:02.111247 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3f756be-04f1-46c8-bd35-2ad9f87d535c None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:02.165557 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:02.217749 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3f756be-04f1-46c8-bd35-2ad9f87d535c None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:02.218276 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6af9f9f0-a8fa-4e24-8ce7-3522cf681b3e None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:02.236548 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:02.255333 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ffad788-80df-465c-80a2-077bf2be8bbb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:02.255598 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3da18309-0c32-4662-82eb-e0fd36992aba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:02.255971 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ffad788-80df-465c-80a2-077bf2be8bbb None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:02.256167 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3da18309-0c32-4662-82eb-e0fd36992aba None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:02.256267 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ffad788-80df-465c-80a2-077bf2be8bbb None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:02.256442 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3da18309-0c32-4662-82eb-e0fd36992aba None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:02.256549 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ffad788-80df-465c-80a2-077bf2be8bbb None admin] PATH_INFO: `/v3/projects/service` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:02.256715 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3da18309-0c32-4662-82eb-e0fd36992aba None admin] PATH_INFO: `/v3/domains` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:02.257380 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3da18309-0c32-4662-82eb-e0fd36992aba None admin] RBAC: Adding query filter params (name=Default) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:02.257667 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3da18309-0c32-4662-82eb-e0fd36992aba None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:02.259088 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3da18309-0c32-4662-82eb-e0fd36992aba None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:02.263831 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ffad788-80df-465c-80a2-077bf2be8bbb None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:02.265420 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ffad788-80df-465c-80a2-077bf2be8bbb None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:02.271956 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 289/582] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Aug 28 00:42:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:02.272816 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3da18309-0c32-4662-82eb-e0fd36992aba None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:02.281938 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-0ffad788-80df-465c-80a2-077bf2be8bbb None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 28 00:42:02.281938 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:02.281938 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:02.281938 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:02.281938 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:02.281938 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:02.281938 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:02.281938 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:02.281938 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:02.281938 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:02.281938 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:02.281938 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:02.281938 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 28 00:42:02.281938 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 28 00:42:02.281938 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 28 00:42:02.283063 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 28 00:42:02.283063 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:02.283063 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:02.283063 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:02.283063 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:02.283063 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:02.283063 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:02.283063 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:02.283063 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:02.283063 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:02.283063 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:02.283063 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:02.283063 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:02.283063 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:02.283063 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:02.283063 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:02.283063 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:02.283939 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 28 00:42:02.283939 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 28 00:42:02.283939 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 28 00:42:02.283939 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 28 00:42:02.283939 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 28 00:42:02.283939 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 28 00:42:02.283939 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 28 00:42:02.283939 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:02.283939 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 294/583] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Aug 28 00:42:02 2022] GET /identity/v3/projects/service => generated 88 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:02.283939 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ffad788-80df-465c-80a2-077bf2be8bbb None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:02.387176 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3da18309-0c32-4662-82eb-e0fd36992aba None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:02.391923 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ffad788-80df-465c-80a2-077bf2be8bbb None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:02.406840 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:02.427541 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec69cc3e-d570-4194-bb73-1200eb541518 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:02.427742 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d12d080a-ab23-492f-9ab4-40423d54182c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:02.428064 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec69cc3e-d570-4194-bb73-1200eb541518 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:02.428275 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d12d080a-ab23-492f-9ab4-40423d54182c None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:02.428373 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec69cc3e-d570-4194-bb73-1200eb541518 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:02.428547 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d12d080a-ab23-492f-9ab4-40423d54182c None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:02.428649 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec69cc3e-d570-4194-bb73-1200eb541518 None admin] PATH_INFO: `/v3/domains` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:02.428820 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d12d080a-ab23-492f-9ab4-40423d54182c None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:02.429249 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec69cc3e-d570-4194-bb73-1200eb541518 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:02.429546 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec69cc3e-d570-4194-bb73-1200eb541518 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:02.430941 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec69cc3e-d570-4194-bb73-1200eb541518 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:02.437773 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d12d080a-ab23-492f-9ab4-40423d54182c None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:02.439274 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d12d080a-ab23-492f-9ab4-40423d54182c None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:02.448451 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 290/584] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Aug 28 00:42:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:02.448451 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec69cc3e-d570-4194-bb73-1200eb541518 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:02.452093 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-d12d080a-ab23-492f-9ab4-40423d54182c None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Aug 28 00:42:02.452093 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:02.452093 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:02.452093 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:02.452093 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:02.452093 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:02.452093 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:02.452093 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:02.452093 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:02.452093 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:02.452093 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:02.452093 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:02.452093 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 28 00:42:02.452093 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 28 00:42:02.452093 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 28 00:42:02.452093 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 28 00:42:02.453135 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:02.453135 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:02.453135 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 28 00:42:02.453135 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 28 00:42:02.453135 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 28 00:42:02.453135 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 28 00:42:02.453135 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:02.453135 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:02.453135 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:02.453135 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:02.453135 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:02.453135 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:02.453135 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:02.453135 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:02.453135 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:02.453135 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:02.453135 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:02.453135 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:02.454089 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:02.454089 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:02.454089 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 28 00:42:02.454089 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 28 00:42:02.454089 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 28 00:42:02.454089 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 28 00:42:02.454089 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 28 00:42:02.454089 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 28 00:42:02.454089 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Aug 28 00:42:02.454089 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:02.454089 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 295/585] 10.222.0.42 () {56 vars in 1102 bytes} [Sun Aug 28 00:42:02 2022] GET /identity/v3/users/neutron => generated 85 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:02.454666 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d12d080a-ab23-492f-9ab4-40423d54182c None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:02.499202 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:02.564480 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec69cc3e-d570-4194-bb73-1200eb541518 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:02.570482 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d12d080a-ab23-492f-9ab4-40423d54182c None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:02.600184 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a06b3244-f4fb-4f92-b714-61f096216952 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:02.600839 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a06b3244-f4fb-4f92-b714-61f096216952 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:02.601153 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a06b3244-f4fb-4f92-b714-61f096216952 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:02.601423 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a06b3244-f4fb-4f92-b714-61f096216952 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:02.608720 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d301716-31f4-4e4d-8ae3-080a5b9571ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:02.609589 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d301716-31f4-4e4d-8ae3-080a5b9571ce None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:02.609872 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d301716-31f4-4e4d-8ae3-080a5b9571ce None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:02.610031 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a06b3244-f4fb-4f92-b714-61f096216952 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:02.610150 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d301716-31f4-4e4d-8ae3-080a5b9571ce None admin] PATH_INFO: `/v3/projects` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:02.610834 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d301716-31f4-4e4d-8ae3-080a5b9571ce None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:02.611139 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d301716-31f4-4e4d-8ae3-080a5b9571ce None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:02.611495 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a06b3244-f4fb-4f92-b714-61f096216952 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:02.612720 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d301716-31f4-4e4d-8ae3-080a5b9571ce None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:02.623870 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-a06b3244-f4fb-4f92-b714-61f096216952 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 28 00:42:02.623870 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:02.623870 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:02.623870 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:02.623870 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:02.623870 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:02.623870 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:02.623870 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:02.623870 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:02.623870 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:02.623870 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:02.623870 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:02.623870 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 28 00:42:02.623870 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 28 00:42:02.623870 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 28 00:42:02.623870 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 28 00:42:02.624906 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:02.624906 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:02.624906 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 28 00:42:02.624906 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 28 00:42:02.624906 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 28 00:42:02.624906 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 28 00:42:02.624906 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:02.624906 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:02.624906 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:02.624906 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:02.624906 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:02.624906 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:02.624906 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:02.624906 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:02.624906 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:02.624906 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:02.624906 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:02.624906 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:02.625866 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:02.625866 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:02.625866 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 28 00:42:02.625866 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 28 00:42:02.625866 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 28 00:42:02.625866 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 28 00:42:02.625866 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 28 00:42:02.625866 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 28 00:42:02.625866 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 28 00:42:02.625866 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:42:02.625866 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 291/586] 10.222.0.42 () {56 vars in 1093 bytes} [Sun Aug 28 00:42:02 2022] GET /identity/v3/users/nova => generated 82 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:02.625866 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a06b3244-f4fb-4f92-b714-61f096216952 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:02.632563 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 296/587] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Aug 28 00:42:02 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:02.633787 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d301716-31f4-4e4d-8ae3-080a5b9571ce None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:02.663987 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:02.735699 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a06b3244-f4fb-4f92-b714-61f096216952 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:02.740106 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:02.746073 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d301716-31f4-4e4d-8ae3-080a5b9571ce None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:02.775627 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f691f499-a129-467f-839b-a186da5a49e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:02.776261 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f691f499-a129-467f-839b-a186da5a49e6 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:02.776533 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f691f499-a129-467f-839b-a186da5a49e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:02.776796 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f691f499-a129-467f-839b-a186da5a49e6 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:02.781776 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25a8cc7b-4c55-478c-9e2e-a0ce2d54b6f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:02.782386 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25a8cc7b-4c55-478c-9e2e-a0ce2d54b6f0 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:02.782664 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25a8cc7b-4c55-478c-9e2e-a0ce2d54b6f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:02.782932 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25a8cc7b-4c55-478c-9e2e-a0ce2d54b6f0 None admin] PATH_INFO: `/v3/users` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:02.783670 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25a8cc7b-4c55-478c-9e2e-a0ce2d54b6f0 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:02.783973 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25a8cc7b-4c55-478c-9e2e-a0ce2d54b6f0 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:02.785470 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25a8cc7b-4c55-478c-9e2e-a0ce2d54b6f0 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:02.785687 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f691f499-a129-467f-839b-a186da5a49e6 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:02.787131 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f691f499-a129-467f-839b-a186da5a49e6 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:02.799301 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-f691f499-a129-467f-839b-a186da5a49e6 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Aug 28 00:42:02.799301 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:02.799301 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:02.799301 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:02.799301 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:02.799301 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:02.799301 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:02.799301 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:02.799301 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:02.799301 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:02.799301 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:02.799301 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:02.799301 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 28 00:42:02.799301 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 28 00:42:02.799301 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 28 00:42:02.799301 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 28 00:42:02.800320 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:02.800320 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:02.800320 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 28 00:42:02.800320 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 28 00:42:02.800320 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 28 00:42:02.800320 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 28 00:42:02.800320 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:02.800320 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:02.800320 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:02.800320 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:02.800320 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:02.800320 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:02.800320 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:02.800320 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:02.800320 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:02.800320 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:02.800320 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:02.800320 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:02.801360 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:02.801360 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:02.801360 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 28 00:42:02.801360 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 28 00:42:02.801360 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 28 00:42:02.801360 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 28 00:42:02.801360 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 28 00:42:02.801360 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 28 00:42:02.801360 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Aug 28 00:42:02.801360 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:42:02.801360 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 292/588] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Aug 28 00:42:02 2022] GET /identity/v3/users/cinder => generated 84 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:02.805713 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f691f499-a129-467f-839b-a186da5a49e6 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:02.805865 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 297/589] 10.222.0.42 () {56 vars in 1139 bytes} [Sun Aug 28 00:42:02 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:02.806738 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25a8cc7b-4c55-478c-9e2e-a0ce2d54b6f0 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:02.918357 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25a8cc7b-4c55-478c-9e2e-a0ce2d54b6f0 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:02.921977 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:02.928051 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f691f499-a129-467f-839b-a186da5a49e6 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:02.956449 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1dcf114-7817-4037-a368-5765800319c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:02.957159 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1dcf114-7817-4037-a368-5765800319c6 None admin] REQUEST_METHOD: `PUT` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:02.957442 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1dcf114-7817-4037-a368-5765800319c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:02.957721 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1dcf114-7817-4037-a368-5765800319c6 None admin] PATH_INFO: `/v3/projects/f53e6fdb369b480fbebb4086a505df67/users/14176b54cf6640abb3b1c14e3824d4f1/roles/348070fea3dd4cc2a6e00e29fe82cc25` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:02.966000 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-111acba2-e5ec-46c3-a33b-372d9819bb56 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:02.966606 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-111acba2-e5ec-46c3-a33b-372d9819bb56 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:02.966880 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-111acba2-e5ec-46c3-a33b-372d9819bb56 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:02.967145 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-111acba2-e5ec-46c3-a33b-372d9819bb56 None admin] PATH_INFO: `/v3/users` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:02.967884 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-111acba2-e5ec-46c3-a33b-372d9819bb56 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:02.968215 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-111acba2-e5ec-46c3-a33b-372d9819bb56 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:02.969995 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-111acba2-e5ec-46c3-a33b-372d9819bb56 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:02.973445 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1dcf114-7817-4037-a368-5765800319c6 None admin] RBAC: Authorizing `identity:create_grant(project_id=f53e6fdb369b480fbebb4086a505df67, user_id=14176b54cf6640abb3b1c14e3824d4f1, role_id=348070fea3dd4cc2a6e00e29fe82cc25)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:02.974889 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1dcf114-7817-4037-a368-5765800319c6 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:02.992826 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 293/590] 10.222.0.42 () {56 vars in 1133 bytes} [Sun Aug 28 00:42:02 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:02.993042 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 298/591] 10.222.0.42 () {58 vars in 1439 bytes} [Sun Aug 28 00:42:02 2022] PUT /identity/v3/projects/f53e6fdb369b480fbebb4086a505df67/users/14176b54cf6640abb3b1c14e3824d4f1/roles/348070fea3dd4cc2a6e00e29fe82cc25 => generated 0 bytes in 182 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:42:02.993142 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-111acba2-e5ec-46c3-a33b-372d9819bb56 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:02.993228 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1dcf114-7817-4037-a368-5765800319c6 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:02.998047 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:03.108013 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1dcf114-7817-4037-a368-5765800319c6 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:03.151152 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-111acba2-e5ec-46c3-a33b-372d9819bb56 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:03.152197 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-817ae8ff-3b9a-4056-aaa0-fc4b88f0770b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:03.152854 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-817ae8ff-3b9a-4056-aaa0-fc4b88f0770b None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:03.153156 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-817ae8ff-3b9a-4056-aaa0-fc4b88f0770b None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:03.153432 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-817ae8ff-3b9a-4056-aaa0-fc4b88f0770b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:03.165844 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-817ae8ff-3b9a-4056-aaa0-fc4b88f0770b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:03.167323 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-817ae8ff-3b9a-4056-aaa0-fc4b88f0770b None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:03.185482 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-817ae8ff-3b9a-4056-aaa0-fc4b88f0770b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:42:03.185482 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:03.185482 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:03.185482 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:03.185482 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:03.185482 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:03.185482 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:03.185482 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:03.185482 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:03.185482 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:03.185482 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:03.185482 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:03.185482 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 28 00:42:03.185482 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 28 00:42:03.185482 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 28 00:42:03.185482 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 28 00:42:03.186528 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:03.186528 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:03.186528 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:03.186528 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:03.186528 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:03.186528 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:03.186528 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:03.186528 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:03.186528 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:03.186528 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:03.186528 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:03.186528 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:03.186528 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:03.186528 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:03.186528 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:03.186528 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:03.186528 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 28 00:42:03.187542 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 28 00:42:03.187542 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:42:03.187542 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:03.187542 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 299/592] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:42:02 2022] GET /identity/v3/domains/Default => generated 87 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:03.188230 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-817ae8ff-3b9a-4056-aaa0-fc4b88f0770b None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:03.203511 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5644ad7a-f0dc-4b4e-9f83-422c872d037e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:03.204134 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5644ad7a-f0dc-4b4e-9f83-422c872d037e None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:03.204414 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5644ad7a-f0dc-4b4e-9f83-422c872d037e None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:03.204689 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5644ad7a-f0dc-4b4e-9f83-422c872d037e None admin] PATH_INFO: `/v3/users` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:03.205803 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5644ad7a-f0dc-4b4e-9f83-422c872d037e None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:03.206126 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5644ad7a-f0dc-4b4e-9f83-422c872d037e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:03.207568 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5644ad7a-f0dc-4b4e-9f83-422c872d037e None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:03.229941 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 294/593] 10.222.0.42 () {56 vars in 1137 bytes} [Sun Aug 28 00:42:02 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 241 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:03.230827 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5644ad7a-f0dc-4b4e-9f83-422c872d037e None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:03.246571 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:03.265862 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:03.321389 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-817ae8ff-3b9a-4056-aaa0-fc4b88f0770b None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:03.359916 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5644ad7a-f0dc-4b4e-9f83-422c872d037e None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:03.367618 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdcf919a-ca2d-4092-afee-5c7eebfabddb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:03.369327 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdcf919a-ca2d-4092-afee-5c7eebfabddb None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:03.369828 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdcf919a-ca2d-4092-afee-5c7eebfabddb None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:03.370319 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdcf919a-ca2d-4092-afee-5c7eebfabddb None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:03.386248 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdcf919a-ca2d-4092-afee-5c7eebfabddb None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:03.388993 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdcf919a-ca2d-4092-afee-5c7eebfabddb None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:03.400652 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3daa3a4d-a2b3-4101-9b7a-9611fdad2d28 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:03.401459 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3daa3a4d-a2b3-4101-9b7a-9611fdad2d28 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:03.401964 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3daa3a4d-a2b3-4101-9b7a-9611fdad2d28 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:03.402472 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3daa3a4d-a2b3-4101-9b7a-9611fdad2d28 None admin] PATH_INFO: `/v3/domains` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:03.403220 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3daa3a4d-a2b3-4101-9b7a-9611fdad2d28 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:03.403617 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3daa3a4d-a2b3-4101-9b7a-9611fdad2d28 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:03.405163 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3daa3a4d-a2b3-4101-9b7a-9611fdad2d28 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:03.409087 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-cdcf919a-ca2d-4092-afee-5c7eebfabddb None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:42:03.409087 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:03.409087 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:03.409087 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:03.409087 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:03.409087 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:03.409087 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:03.409087 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:03.409087 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:03.409087 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:03.409087 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:03.409087 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:03.409087 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 28 00:42:03.409087 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 28 00:42:03.409087 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 28 00:42:03.409087 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 28 00:42:03.410252 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:03.410252 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:03.410252 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:03.410252 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:03.410252 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:03.410252 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:03.410252 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:03.410252 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:03.410252 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:03.410252 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:03.410252 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:03.410252 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:03.410252 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:03.410252 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:03.410252 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:03.410252 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:03.410252 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 28 00:42:03.411356 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 28 00:42:03.411356 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:42:03.411356 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:03.413165 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 300/594] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:42:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 225 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:03.414127 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdcf919a-ca2d-4092-afee-5c7eebfabddb None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:03.429065 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 295/595] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Aug 28 00:42:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:03.430490 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3daa3a4d-a2b3-4101-9b7a-9611fdad2d28 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:03.519848 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:03.550012 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdcf919a-ca2d-4092-afee-5c7eebfabddb None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:03.557559 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3daa3a4d-a2b3-4101-9b7a-9611fdad2d28 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:03.569329 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:03.589538 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-077e0068-1874-43b3-a2d5-cf59d8c47605 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:03.590185 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-077e0068-1874-43b3-a2d5-cf59d8c47605 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:03.590466 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-077e0068-1874-43b3-a2d5-cf59d8c47605 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:03.590740 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-077e0068-1874-43b3-a2d5-cf59d8c47605 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:03.602725 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-077e0068-1874-43b3-a2d5-cf59d8c47605 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:03.602950 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9797ccc2-7251-4e10-af3d-9669a35b8627 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:03.603689 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9797ccc2-7251-4e10-af3d-9669a35b8627 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:03.604068 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9797ccc2-7251-4e10-af3d-9669a35b8627 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:03.604185 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-077e0068-1874-43b3-a2d5-cf59d8c47605 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:03.604505 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9797ccc2-7251-4e10-af3d-9669a35b8627 None admin] PATH_INFO: `/v3/domains` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:03.605274 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9797ccc2-7251-4e10-af3d-9669a35b8627 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:03.605672 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9797ccc2-7251-4e10-af3d-9669a35b8627 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:03.607864 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9797ccc2-7251-4e10-af3d-9669a35b8627 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:03.619537 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-077e0068-1874-43b3-a2d5-cf59d8c47605 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:42:03.619537 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:03.619537 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:03.619537 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:03.619537 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:03.619537 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:03.619537 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:03.619537 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:03.619537 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:03.619537 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:03.619537 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:03.619537 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:03.619537 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 28 00:42:03.619537 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 28 00:42:03.619537 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 28 00:42:03.619537 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 28 00:42:03.620526 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:03.620526 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:03.620526 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:03.620526 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:03.620526 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:03.620526 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:03.620526 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:03.620526 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:03.620526 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:03.620526 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:03.620526 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:03.620526 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:03.620526 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:03.620526 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:03.620526 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:03.620526 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:03.620526 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 28 00:42:03.621431 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 28 00:42:03.621431 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:42:03.621431 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:03.625795 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 301/596] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:42:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:03.626729 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-077e0068-1874-43b3-a2d5-cf59d8c47605 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:03.627277 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 296/597] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Aug 28 00:42:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:03.628191 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9797ccc2-7251-4e10-af3d-9669a35b8627 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:03.744818 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-077e0068-1874-43b3-a2d5-cf59d8c47605 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:03.759638 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9797ccc2-7251-4e10-af3d-9669a35b8627 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:03.786912 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95bf2202-4c8d-4271-86a3-808a5fd848eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:03.787498 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95bf2202-4c8d-4271-86a3-808a5fd848eb None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:03.787778 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95bf2202-4c8d-4271-86a3-808a5fd848eb None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:03.788052 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95bf2202-4c8d-4271-86a3-808a5fd848eb None admin] PATH_INFO: `/v3/projects/service` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:03.791784 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:03.796954 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95bf2202-4c8d-4271-86a3-808a5fd848eb None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:03.797458 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85580de8-d589-4e1b-a7d5-87293b81f9e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:03.798071 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85580de8-d589-4e1b-a7d5-87293b81f9e2 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:03.798350 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85580de8-d589-4e1b-a7d5-87293b81f9e2 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:03.798447 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95bf2202-4c8d-4271-86a3-808a5fd848eb None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:03.798682 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85580de8-d589-4e1b-a7d5-87293b81f9e2 None admin] PATH_INFO: `/v3/domains` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:03.799312 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85580de8-d589-4e1b-a7d5-87293b81f9e2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:03.799766 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85580de8-d589-4e1b-a7d5-87293b81f9e2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:03.801220 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85580de8-d589-4e1b-a7d5-87293b81f9e2 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:03.810419 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-95bf2202-4c8d-4271-86a3-808a5fd848eb None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 28 00:42:03.810419 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:03.810419 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:03.810419 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:03.810419 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:03.810419 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:03.810419 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:03.810419 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:03.810419 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:03.810419 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:03.810419 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:03.810419 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:03.810419 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 28 00:42:03.810419 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 28 00:42:03.810419 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 28 00:42:03.811378 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 28 00:42:03.811378 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:03.811378 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:03.811378 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:03.811378 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:03.811378 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:03.811378 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:03.811378 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:03.811378 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:03.811378 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:03.811378 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:03.811378 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:03.811378 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:03.811378 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:03.811378 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:03.811378 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:03.811378 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:03.812556 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 28 00:42:03.812556 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 28 00:42:03.812556 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 28 00:42:03.812556 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 28 00:42:03.812556 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 28 00:42:03.812556 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 28 00:42:03.812556 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 28 00:42:03.812556 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:03.818740 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 297/598] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Aug 28 00:42:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:03.818740 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85580de8-d589-4e1b-a7d5-87293b81f9e2 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:03.819123 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 302/599] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Aug 28 00:42:03 2022] GET /identity/v3/projects/service => generated 88 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:03.819123 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94207807-ba7c-4ec6-92a1-7bc2b722d7d4 None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:03.819123 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94207807-ba7c-4ec6-92a1-7bc2b722d7d4 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:03.819123 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94207807-ba7c-4ec6-92a1-7bc2b722d7d4 None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:03.820100 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 303/600] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:42:03 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:42:03.820782 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94207807-ba7c-4ec6-92a1-7bc2b722d7d4 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:03.845141 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:03.924845 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85580de8-d589-4e1b-a7d5-87293b81f9e2 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:03.931382 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94207807-ba7c-4ec6-92a1-7bc2b722d7d4 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:03.968109 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c909a9bf-db51-4e75-a1c1-61a2d53b8862 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:03.969204 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c909a9bf-db51-4e75-a1c1-61a2d53b8862 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:03.969693 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c909a9bf-db51-4e75-a1c1-61a2d53b8862 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:03.970159 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c909a9bf-db51-4e75-a1c1-61a2d53b8862 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:03.970851 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cbf2604-88a3-4f21-a98c-aef011d313d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:03.971638 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cbf2604-88a3-4f21-a98c-aef011d313d5 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:03.972078 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cbf2604-88a3-4f21-a98c-aef011d313d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:03.972505 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cbf2604-88a3-4f21-a98c-aef011d313d5 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:03.979738 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cbf2604-88a3-4f21-a98c-aef011d313d5 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:03.981518 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cbf2604-88a3-4f21-a98c-aef011d313d5 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:03.981959 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c909a9bf-db51-4e75-a1c1-61a2d53b8862 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:03.984655 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c909a9bf-db51-4e75-a1c1-61a2d53b8862 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:03.993433 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-5cbf2604-88a3-4f21-a98c-aef011d313d5 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 28 00:42:03.993433 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:03.993433 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:03.993433 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:03.993433 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:03.993433 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:03.993433 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:03.993433 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:03.993433 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:03.993433 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:03.993433 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:03.993433 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:03.993433 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 28 00:42:03.993433 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 28 00:42:03.993433 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 28 00:42:03.994492 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 28 00:42:03.994492 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:03.994492 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:03.994492 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:03.994492 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:03.994492 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:03.994492 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:03.994492 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:03.994492 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:03.994492 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:03.994492 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:03.994492 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:03.994492 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:03.994492 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:03.994492 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:03.994492 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:03.994492 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:03.995613 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 28 00:42:03.995613 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 28 00:42:03.995613 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 28 00:42:03.995613 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 28 00:42:03.995613 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 28 00:42:03.995613 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 28 00:42:03.995613 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 28 00:42:03.995613 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:03.997338 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 304/601] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Aug 28 00:42:03 2022] GET /identity/v3/projects/service => generated 88 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:03.998295 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cbf2604-88a3-4f21-a98c-aef011d313d5 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:04.005928 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-c909a9bf-db51-4e75-a1c1-61a2d53b8862 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 28 00:42:04.005928 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:04.005928 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:04.005928 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:04.005928 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:04.005928 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:04.005928 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:04.005928 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:04.005928 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:04.005928 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:04.005928 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:04.005928 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:04.005928 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 28 00:42:04.005928 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 28 00:42:04.005928 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 28 00:42:04.006967 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 28 00:42:04.006967 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:04.006967 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:04.006967 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:04.006967 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:04.006967 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:04.006967 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:04.006967 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:04.006967 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:04.006967 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:04.006967 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:04.006967 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:04.006967 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:04.006967 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:04.006967 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:04.006967 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:04.006967 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:04.007973 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 28 00:42:04.007973 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 28 00:42:04.007973 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 28 00:42:04.007973 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 28 00:42:04.007973 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 28 00:42:04.007973 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 28 00:42:04.007973 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 28 00:42:04.007973 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:42:04.007973 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 298/602] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Aug 28 00:42:03 2022] GET /identity/v3/projects/service => generated 88 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:04.009634 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03896a6b-42c6-490d-b70c-8fd611270a6b None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:04.010135 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03896a6b-42c6-490d-b70c-8fd611270a6b None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:04.010627 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03896a6b-42c6-490d-b70c-8fd611270a6b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:04.039193 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-03896a6b-42c6-490d-b70c-8fd611270a6b None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:42:04.040414 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-03896a6b-42c6-490d-b70c-8fd611270a6b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:42:04.093888 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:04.096971 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:04.123918 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cbf2604-88a3-4f21-a98c-aef011d313d5 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:04.130426 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 299/603] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:42:04 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 28 00:42:04.131669 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03896a6b-42c6-490d-b70c-8fd611270a6b None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:04.162103 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79b2e349-4fb6-408d-a0b6-2af7b20b6339 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:04.162733 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b2e349-4fb6-408d-a0b6-2af7b20b6339 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:04.163014 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b2e349-4fb6-408d-a0b6-2af7b20b6339 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:04.163285 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b2e349-4fb6-408d-a0b6-2af7b20b6339 None admin] PATH_INFO: `/v3/projects` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:04.164552 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79b2e349-4fb6-408d-a0b6-2af7b20b6339 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:04.164552 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79b2e349-4fb6-408d-a0b6-2af7b20b6339 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:04.165950 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79b2e349-4fb6-408d-a0b6-2af7b20b6339 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:04.180418 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 305/604] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Aug 28 00:42:03 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:04.181383 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79b2e349-4fb6-408d-a0b6-2af7b20b6339 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:04.258187 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03896a6b-42c6-490d-b70c-8fd611270a6b None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:04.300231 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79b2e349-4fb6-408d-a0b6-2af7b20b6339 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:04.301834 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04085cc6-5f75-4b72-bd64-b1399eccd86c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:04.303539 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04085cc6-5f75-4b72-bd64-b1399eccd86c None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:04.304018 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04085cc6-5f75-4b72-bd64-b1399eccd86c None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:04.304483 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04085cc6-5f75-4b72-bd64-b1399eccd86c None admin] PATH_INFO: `/v3/projects` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:04.305397 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04085cc6-5f75-4b72-bd64-b1399eccd86c None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:04.305933 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04085cc6-5f75-4b72-bd64-b1399eccd86c None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:04.307563 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04085cc6-5f75-4b72-bd64-b1399eccd86c None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:04.323115 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 300/605] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Aug 28 00:42:04 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:04.324391 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04085cc6-5f75-4b72-bd64-b1399eccd86c None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:04.343408 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e9ea513-21e5-4c05-90cf-5eb57418f4d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:04.344107 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e9ea513-21e5-4c05-90cf-5eb57418f4d0 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:04.344304 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e9ea513-21e5-4c05-90cf-5eb57418f4d0 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:04.344573 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e9ea513-21e5-4c05-90cf-5eb57418f4d0 None admin] PATH_INFO: `/v3/projects` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:04.345309 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e9ea513-21e5-4c05-90cf-5eb57418f4d0 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:04.345751 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e9ea513-21e5-4c05-90cf-5eb57418f4d0 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:04.347057 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e9ea513-21e5-4c05-90cf-5eb57418f4d0 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:04.357276 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:04.365351 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 306/606] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Aug 28 00:42:04 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:04.366221 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e9ea513-21e5-4c05-90cf-5eb57418f4d0 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:04.385951 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:04.451679 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04085cc6-5f75-4b72-bd64-b1399eccd86c None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:04.486346 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e9ea513-21e5-4c05-90cf-5eb57418f4d0 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:04.492833 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59ab6ff8-005f-4050-923d-522fcc6901d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:04.493555 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59ab6ff8-005f-4050-923d-522fcc6901d7 None admin] REQUEST_METHOD: `PUT` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:04.493838 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59ab6ff8-005f-4050-923d-522fcc6901d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:04.494108 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59ab6ff8-005f-4050-923d-522fcc6901d7 None admin] PATH_INFO: `/v3/projects/f53e6fdb369b480fbebb4086a505df67/users/364a28c108e744728917ba2b0aa2347d/roles/348070fea3dd4cc2a6e00e29fe82cc25` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:04.509329 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59ab6ff8-005f-4050-923d-522fcc6901d7 None admin] RBAC: Authorizing `identity:create_grant(project_id=f53e6fdb369b480fbebb4086a505df67, user_id=364a28c108e744728917ba2b0aa2347d, role_id=348070fea3dd4cc2a6e00e29fe82cc25)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:04.510822 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59ab6ff8-005f-4050-923d-522fcc6901d7 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:04.525337 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 301/607] 10.222.0.42 () {58 vars in 1439 bytes} [Sun Aug 28 00:42:04 2022] PUT /identity/v3/projects/f53e6fdb369b480fbebb4086a505df67/users/364a28c108e744728917ba2b0aa2347d/roles/348070fea3dd4cc2a6e00e29fe82cc25 => generated 0 bytes in 201 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:42:04.525719 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc6396cb-70c4-44ff-b559-fe3fc793f0e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:04.526139 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59ab6ff8-005f-4050-923d-522fcc6901d7 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:04.526864 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc6396cb-70c4-44ff-b559-fe3fc793f0e5 None admin] REQUEST_METHOD: `PUT` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:04.527335 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc6396cb-70c4-44ff-b559-fe3fc793f0e5 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:04.527812 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc6396cb-70c4-44ff-b559-fe3fc793f0e5 None admin] PATH_INFO: `/v3/projects/f53e6fdb369b480fbebb4086a505df67/users/879afc99ebb9402b94fadc6091040c5c/roles/348070fea3dd4cc2a6e00e29fe82cc25` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:04.549950 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc6396cb-70c4-44ff-b559-fe3fc793f0e5 None admin] RBAC: Authorizing `identity:create_grant(project_id=f53e6fdb369b480fbebb4086a505df67, user_id=879afc99ebb9402b94fadc6091040c5c, role_id=348070fea3dd4cc2a6e00e29fe82cc25)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:04.552164 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc6396cb-70c4-44ff-b559-fe3fc793f0e5 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:04.573647 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 307/608] 10.222.0.42 () {58 vars in 1439 bytes} [Sun Aug 28 00:42:04 2022] PUT /identity/v3/projects/f53e6fdb369b480fbebb4086a505df67/users/879afc99ebb9402b94fadc6091040c5c/roles/348070fea3dd4cc2a6e00e29fe82cc25 => generated 0 bytes in 207 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:42:04.574992 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc6396cb-70c4-44ff-b559-fe3fc793f0e5 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:04.696467 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:04.718535 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:04.734078 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59ab6ff8-005f-4050-923d-522fcc6901d7 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:04.761995 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc6396cb-70c4-44ff-b559-fe3fc793f0e5 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:04.788369 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df4f168a-85ae-4474-af95-9ecc2b37a63d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:04.789651 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df4f168a-85ae-4474-af95-9ecc2b37a63d None admin] REQUEST_METHOD: `PUT` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:04.790076 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df4f168a-85ae-4474-af95-9ecc2b37a63d None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:04.790493 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df4f168a-85ae-4474-af95-9ecc2b37a63d None admin] PATH_INFO: `/v3/projects/f53e6fdb369b480fbebb4086a505df67/users/168b07df50c24593ba73dbde5b037aed/roles/348070fea3dd4cc2a6e00e29fe82cc25` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:04.804446 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b04603a7-5d40-45df-97c8-22b5ec2e9a3e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:04.805236 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b04603a7-5d40-45df-97c8-22b5ec2e9a3e None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:04.805603 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b04603a7-5d40-45df-97c8-22b5ec2e9a3e None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:04.805969 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b04603a7-5d40-45df-97c8-22b5ec2e9a3e None admin] PATH_INFO: `/v3/roles/service` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:04.811983 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df4f168a-85ae-4474-af95-9ecc2b37a63d None admin] RBAC: Authorizing `identity:create_grant(project_id=f53e6fdb369b480fbebb4086a505df67, user_id=168b07df50c24593ba73dbde5b037aed, role_id=348070fea3dd4cc2a6e00e29fe82cc25)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:04.814196 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df4f168a-85ae-4474-af95-9ecc2b37a63d None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:04.817903 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b04603a7-5d40-45df-97c8-22b5ec2e9a3e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 28 00:42:04.823231 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-b04603a7-5d40-45df-97c8-22b5ec2e9a3e None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 28 00:42:04.823231 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:04.823231 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:04.823231 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:04.823231 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:04.823231 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:04.823231 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:04.823231 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:04.823231 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:04.823231 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:04.823231 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:04.823231 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:04.823231 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 28 00:42:04.823231 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 28 00:42:04.823231 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 28 00:42:04.823231 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 28 00:42:04.824655 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 28 00:42:04.824655 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 28 00:42:04.824655 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 28 00:42:04.824655 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 28 00:42:04.824655 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:04.824655 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:04.824655 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:04.824655 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:04.824655 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:04.824655 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:04.824655 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:04.824655 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:04.824655 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:04.824655 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:04.824655 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:04.824655 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:04.825778 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:04.825778 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:04.825778 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:04.825778 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:04.825778 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 28 00:42:04.825778 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 28 00:42:04.825778 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 28 00:42:04.825778 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 28 00:42:04.825778 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 28 00:42:04.825778 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 28 00:42:04.825778 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 28 00:42:04.825778 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:04.826707 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 308/609] 10.222.0.42 () {56 vars in 1102 bytes} [Sun Aug 28 00:42:04 2022] GET /identity/v3/roles/service => generated 85 bytes in 252 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:04.833243 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b04603a7-5d40-45df-97c8-22b5ec2e9a3e None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:04.835457 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 302/610] 10.222.0.42 () {58 vars in 1439 bytes} [Sun Aug 28 00:42:04 2022] PUT /identity/v3/projects/f53e6fdb369b480fbebb4086a505df67/users/168b07df50c24593ba73dbde5b037aed/roles/348070fea3dd4cc2a6e00e29fe82cc25 => generated 0 bytes in 309 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:42:04.966756 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b04603a7-5d40-45df-97c8-22b5ec2e9a3e None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:04.979898 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:05.007502 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bff5922b-c6ea-4ede-b79e-9b2d4bd88e40 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:05.013883 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff5922b-c6ea-4ede-b79e-9b2d4bd88e40 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:05.014222 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff5922b-c6ea-4ede-b79e-9b2d4bd88e40 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:05.014433 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff5922b-c6ea-4ede-b79e-9b2d4bd88e40 None admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:05.015198 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bff5922b-c6ea-4ede-b79e-9b2d4bd88e40 None admin] RBAC: Adding query filter params (name=service) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:05.015505 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bff5922b-c6ea-4ede-b79e-9b2d4bd88e40 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:05.016921 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bff5922b-c6ea-4ede-b79e-9b2d4bd88e40 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:05.029168 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 309/611] 10.222.0.42 () {56 vars in 1103 bytes} [Sun Aug 28 00:42:04 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 28 00:42:05.033479 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff1dd300-6e05-44e8-83e9-e83b0e97a368 None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:05.034007 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff1dd300-6e05-44e8-83e9-e83b0e97a368 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:05.034258 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff1dd300-6e05-44e8-83e9-e83b0e97a368 None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:05.034761 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bff5922b-c6ea-4ede-b79e-9b2d4bd88e40 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:05.035940 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 303/612] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:42:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:42:05.041610 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-959241e2-acbc-4281-be07-430bb698fcf2 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:05.041910 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-959241e2-acbc-4281-be07-430bb698fcf2 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:05.042157 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-959241e2-acbc-4281-be07-430bb698fcf2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:05.070099 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-959241e2-acbc-4281-be07-430bb698fcf2 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:42:05.071269 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-959241e2-acbc-4281-be07-430bb698fcf2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:42:05.169632 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 304/613] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:42:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 28 00:42:05.178915 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bff5922b-c6ea-4ede-b79e-9b2d4bd88e40 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:05.219073 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e8acc6f-174c-460c-a791-4698f15eecf3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:05.219726 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e8acc6f-174c-460c-a791-4698f15eecf3 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:05.220038 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e8acc6f-174c-460c-a791-4698f15eecf3 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:05.220314 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e8acc6f-174c-460c-a791-4698f15eecf3 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:05.232304 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e8acc6f-174c-460c-a791-4698f15eecf3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:05.233937 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e8acc6f-174c-460c-a791-4698f15eecf3 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:05.249484 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-0e8acc6f-174c-460c-a791-4698f15eecf3 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:42:05.249484 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:05.249484 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:05.249484 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:05.249484 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:05.249484 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:05.249484 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:05.249484 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:05.249484 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:05.249484 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:05.249484 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:05.249484 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:05.249484 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 28 00:42:05.249484 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 28 00:42:05.249484 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 28 00:42:05.249484 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 28 00:42:05.250469 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:05.250469 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:05.250469 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:05.250469 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:05.250469 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:05.250469 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:05.250469 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:05.250469 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:05.250469 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:05.250469 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:05.250469 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:05.250469 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:05.250469 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:05.250469 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:05.250469 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:05.250469 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:05.250469 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 28 00:42:05.251982 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 28 00:42:05.251982 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:42:05.251982 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:05.251982 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 310/614] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:42:05 2022] GET /identity/v3/domains/Default => generated 87 bytes in 217 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 28 00:42:05.256350 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-959241e2-acbc-4281-be07-430bb698fcf2 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:05.347901 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:05.386838 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-959241e2-acbc-4281-be07-430bb698fcf2 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:05.429822 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50aa8897-c584-4968-bcbf-c411b33bfc49 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:05.430437 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50aa8897-c584-4968-bcbf-c411b33bfc49 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:05.430713 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50aa8897-c584-4968-bcbf-c411b33bfc49 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:05.430982 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50aa8897-c584-4968-bcbf-c411b33bfc49 None admin] PATH_INFO: `/v3/domains` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:05.431622 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50aa8897-c584-4968-bcbf-c411b33bfc49 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:05.431922 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50aa8897-c584-4968-bcbf-c411b33bfc49 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:05.433449 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50aa8897-c584-4968-bcbf-c411b33bfc49 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:05.448117 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 305/615] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Aug 28 00:42:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:05.453900 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e8acc6f-174c-460c-a791-4698f15eecf3 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:05.481648 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:05.565429 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e8acc6f-174c-460c-a791-4698f15eecf3 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:05.604561 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10a31abf-d1c2-4210-b107-18abc5c7ec24 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:05.605149 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10a31abf-d1c2-4210-b107-18abc5c7ec24 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:05.605426 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10a31abf-d1c2-4210-b107-18abc5c7ec24 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:05.605696 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10a31abf-d1c2-4210-b107-18abc5c7ec24 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:05.615678 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10a31abf-d1c2-4210-b107-18abc5c7ec24 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:05.617189 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10a31abf-d1c2-4210-b107-18abc5c7ec24 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:05.631602 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-10a31abf-d1c2-4210-b107-18abc5c7ec24 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Aug 28 00:42:05.631602 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:05.631602 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:05.631602 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:05.631602 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:05.631602 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:05.631602 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:05.631602 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:05.631602 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:05.631602 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:05.631602 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:05.631602 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:05.631602 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 28 00:42:05.631602 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 28 00:42:05.631602 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 28 00:42:05.631602 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 28 00:42:05.632595 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:05.632595 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:05.632595 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 28 00:42:05.632595 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 28 00:42:05.632595 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 28 00:42:05.632595 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 28 00:42:05.632595 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:05.632595 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:05.632595 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:05.632595 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:05.632595 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:05.632595 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:05.632595 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:05.632595 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:05.632595 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:05.632595 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:05.632595 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:05.632595 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:05.633607 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:05.633607 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:05.633607 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 28 00:42:05.633607 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 28 00:42:05.633607 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 28 00:42:05.633607 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 28 00:42:05.633607 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 28 00:42:05.633607 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 28 00:42:05.633607 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Aug 28 00:42:05.633607 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:05.633607 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 311/616] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Aug 28 00:42:05 2022] GET /identity/v3/users/swift => generated 83 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:05.642062 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50aa8897-c584-4968-bcbf-c411b33bfc49 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:05.658161 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10a31abf-d1c2-4210-b107-18abc5c7ec24 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:05.795581 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:05.814187 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10a31abf-d1c2-4210-b107-18abc5c7ec24 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:05.850546 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c700252-d6fa-426d-b48f-cc26f2e3a12f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:05.851550 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c700252-d6fa-426d-b48f-cc26f2e3a12f None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:05.852102 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c700252-d6fa-426d-b48f-cc26f2e3a12f None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:05.852567 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c700252-d6fa-426d-b48f-cc26f2e3a12f None admin] PATH_INFO: `/v3/roles/service` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:05.864241 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c700252-d6fa-426d-b48f-cc26f2e3a12f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 28 00:42:05.868782 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50aa8897-c584-4968-bcbf-c411b33bfc49 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:05.870217 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-5c700252-d6fa-426d-b48f-cc26f2e3a12f None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 28 00:42:05.870217 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:05.870217 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:05.870217 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:05.870217 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:05.870217 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:05.870217 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:05.870217 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:05.870217 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:05.870217 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:05.870217 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:05.870217 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:05.870217 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 28 00:42:05.870217 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 28 00:42:05.870217 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 28 00:42:05.870217 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 28 00:42:05.871126 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 28 00:42:05.871126 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 28 00:42:05.871126 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 28 00:42:05.871126 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 28 00:42:05.871126 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:05.871126 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:05.871126 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:05.871126 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:05.871126 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:05.871126 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:05.871126 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:05.871126 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:05.871126 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:05.871126 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:05.871126 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:05.871126 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:05.871944 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:05.871944 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:05.871944 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:05.871944 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:05.871944 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 28 00:42:05.871944 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 28 00:42:05.871944 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 28 00:42:05.871944 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 28 00:42:05.871944 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 28 00:42:05.871944 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 28 00:42:05.871944 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 28 00:42:05.871944 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:05.874598 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 312/617] 10.222.0.42 () {56 vars in 1102 bytes} [Sun Aug 28 00:42:05 2022] GET /identity/v3/roles/service => generated 85 bytes in 217 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 28 00:42:05.879943 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c700252-d6fa-426d-b48f-cc26f2e3a12f None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:05.943736 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:05.959060 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7c188fa-7714-49ca-9ecd-df2eef96c5e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:05.959683 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7c188fa-7714-49ca-9ecd-df2eef96c5e5 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:05.959964 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7c188fa-7714-49ca-9ecd-df2eef96c5e5 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:05.960237 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7c188fa-7714-49ca-9ecd-df2eef96c5e5 None admin] PATH_INFO: `/v3/users` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:05.960990 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7c188fa-7714-49ca-9ecd-df2eef96c5e5 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:05.961301 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7c188fa-7714-49ca-9ecd-df2eef96c5e5 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:05.962758 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7c188fa-7714-49ca-9ecd-df2eef96c5e5 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:05.997856 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 306/618] 10.222.0.42 () {56 vars in 1135 bytes} [Sun Aug 28 00:42:05 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 356 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:06.005358 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7c188fa-7714-49ca-9ecd-df2eef96c5e5 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:06.005943 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c700252-d6fa-426d-b48f-cc26f2e3a12f None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:06.043434 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c65fc49-e76d-477f-9d28-d7e289e2a23e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:06.044593 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c65fc49-e76d-477f-9d28-d7e289e2a23e None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:06.045147 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c65fc49-e76d-477f-9d28-d7e289e2a23e None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:06.045642 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c65fc49-e76d-477f-9d28-d7e289e2a23e None admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:06.046587 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c65fc49-e76d-477f-9d28-d7e289e2a23e None admin] RBAC: Adding query filter params (name=service) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:06.047098 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c65fc49-e76d-477f-9d28-d7e289e2a23e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:06.048901 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c65fc49-e76d-477f-9d28-d7e289e2a23e None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:06.058967 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 313/619] 10.222.0.42 () {56 vars in 1103 bytes} [Sun Aug 28 00:42:05 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:06.068513 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c65fc49-e76d-477f-9d28-d7e289e2a23e None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:06.083631 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:06.144360 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7c188fa-7714-49ca-9ecd-df2eef96c5e5 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:06.184427 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d06ad969-65f8-4b75-9faf-2123ef63b85e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:06.185551 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d06ad969-65f8-4b75-9faf-2123ef63b85e None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:06.186194 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d06ad969-65f8-4b75-9faf-2123ef63b85e None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:06.186776 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d06ad969-65f8-4b75-9faf-2123ef63b85e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:06.199259 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d06ad969-65f8-4b75-9faf-2123ef63b85e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:06.201064 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d06ad969-65f8-4b75-9faf-2123ef63b85e None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:06.219615 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-d06ad969-65f8-4b75-9faf-2123ef63b85e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:42:06.219615 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:06.219615 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:06.219615 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:06.219615 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:06.219615 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:06.219615 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:06.219615 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:06.219615 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:06.219615 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:06.219615 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:06.219615 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:06.219615 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 28 00:42:06.219615 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 28 00:42:06.219615 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 28 00:42:06.219615 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 28 00:42:06.220628 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:06.220628 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:06.220628 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:06.220628 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:06.220628 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:06.220628 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:06.220628 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:06.220628 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:06.220628 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:06.220628 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:06.220628 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:06.220628 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:06.220628 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:06.220628 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:06.220628 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:06.220628 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:06.220628 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 28 00:42:06.221573 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 28 00:42:06.221573 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:42:06.221573 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:42:06.224013 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 307/620] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:42:06 2022] GET /identity/v3/domains/Default => generated 87 bytes in 219 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 28 00:42:06.229800 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d06ad969-65f8-4b75-9faf-2123ef63b85e None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:06.299946 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:06.328279 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c65fc49-e76d-477f-9d28-d7e289e2a23e None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:06.393312 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d06ad969-65f8-4b75-9faf-2123ef63b85e None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:06.397945 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10b7401e-7183-4f4c-9788-bb2ccba1c7dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:06.398922 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10b7401e-7183-4f4c-9788-bb2ccba1c7dc None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:06.399517 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10b7401e-7183-4f4c-9788-bb2ccba1c7dc None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:06.400748 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10b7401e-7183-4f4c-9788-bb2ccba1c7dc None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:06.425967 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10b7401e-7183-4f4c-9788-bb2ccba1c7dc None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:06.428556 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10b7401e-7183-4f4c-9788-bb2ccba1c7dc None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:06.455877 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-10b7401e-7183-4f4c-9788-bb2ccba1c7dc None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:42:06.455877 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:06.455877 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:06.455877 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:06.455877 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:06.455877 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:06.455877 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:06.455877 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:06.455877 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:06.455877 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:06.455877 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:06.455877 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:06.455877 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 28 00:42:06.455877 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 28 00:42:06.455877 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 28 00:42:06.455877 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 28 00:42:06.456900 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:06.456900 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:06.456900 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:06.456900 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:06.456900 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:06.456900 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:06.456900 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:06.456900 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:06.456900 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:06.456900 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:06.456900 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:06.456900 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:06.456900 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:06.456900 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:06.456900 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:06.456900 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:06.456900 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 28 00:42:06.457823 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 28 00:42:06.457823 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:42:06.457823 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:06.462931 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 314/621] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:42:06 2022] GET /identity/v3/domains/Default => generated 87 bytes in 395 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:06.467823 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-507c9774-304a-4dfd-84de-3e3af082324a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:06.469017 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10b7401e-7183-4f4c-9788-bb2ccba1c7dc None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:06.473794 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-507c9774-304a-4dfd-84de-3e3af082324a None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:06.474054 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-507c9774-304a-4dfd-84de-3e3af082324a None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:06.474338 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-507c9774-304a-4dfd-84de-3e3af082324a None admin] PATH_INFO: `/v3/domains` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:06.475016 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-507c9774-304a-4dfd-84de-3e3af082324a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:06.475318 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-507c9774-304a-4dfd-84de-3e3af082324a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:06.476728 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-507c9774-304a-4dfd-84de-3e3af082324a None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:06.503536 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 308/622] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Aug 28 00:42:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 274 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:06.508736 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-507c9774-304a-4dfd-84de-3e3af082324a None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:06.537716 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:06.586268 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10b7401e-7183-4f4c-9788-bb2ccba1c7dc None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:06.624087 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eb80ffc-1fec-4504-a983-52b8644109f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:06.625208 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eb80ffc-1fec-4504-a983-52b8644109f8 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:06.625714 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eb80ffc-1fec-4504-a983-52b8644109f8 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:06.626234 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eb80ffc-1fec-4504-a983-52b8644109f8 None admin] PATH_INFO: `/v3/domains` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:06.627154 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8eb80ffc-1fec-4504-a983-52b8644109f8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:06.627653 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8eb80ffc-1fec-4504-a983-52b8644109f8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:06.629263 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8eb80ffc-1fec-4504-a983-52b8644109f8 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:06.643422 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 315/623] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Aug 28 00:42:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:06.649895 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eb80ffc-1fec-4504-a983-52b8644109f8 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:06.746623 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-507c9774-304a-4dfd-84de-3e3af082324a None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:06.748829 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:06.812348 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2859fb9-7f2b-4520-8632-f958c02b556d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:06.815056 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2859fb9-7f2b-4520-8632-f958c02b556d None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:06.815381 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2859fb9-7f2b-4520-8632-f958c02b556d None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:06.815654 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2859fb9-7f2b-4520-8632-f958c02b556d None admin] PATH_INFO: `/v3/projects/service` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:06.830845 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eb80ffc-1fec-4504-a983-52b8644109f8 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:06.833338 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2859fb9-7f2b-4520-8632-f958c02b556d None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:06.834882 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2859fb9-7f2b-4520-8632-f958c02b556d None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:06.859281 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-e2859fb9-7f2b-4520-8632-f958c02b556d None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 28 00:42:06.859281 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:06.859281 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:06.859281 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:06.859281 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:06.859281 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:06.859281 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:06.859281 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:06.859281 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:06.859281 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:06.859281 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:06.859281 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:06.859281 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 28 00:42:06.859281 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 28 00:42:06.859281 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 28 00:42:06.860235 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 28 00:42:06.860235 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:06.860235 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:06.860235 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:06.860235 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:06.860235 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:06.860235 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:06.860235 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:06.860235 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:06.860235 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:06.860235 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:06.860235 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:06.860235 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:06.860235 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:06.860235 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:06.860235 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:06.860235 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:06.861161 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 28 00:42:06.861161 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 28 00:42:06.861161 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 28 00:42:06.861161 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 28 00:42:06.861161 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 28 00:42:06.861161 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 28 00:42:06.861161 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 28 00:42:06.861161 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:42:06.861613 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:06.867956 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 309/624] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Aug 28 00:42:06 2022] GET /identity/v3/projects/service => generated 88 bytes in 360 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 28 00:42:06.874669 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2859fb9-7f2b-4520-8632-f958c02b556d None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:06.899334 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f69f083-84ff-4884-9516-113df0349e32 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:06.899898 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f69f083-84ff-4884-9516-113df0349e32 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:06.900178 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f69f083-84ff-4884-9516-113df0349e32 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:06.900452 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f69f083-84ff-4884-9516-113df0349e32 None admin] PATH_INFO: `/v3/users/glance` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:06.921856 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f69f083-84ff-4884-9516-113df0349e32 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:06.923435 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f69f083-84ff-4884-9516-113df0349e32 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:06.946699 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-2f69f083-84ff-4884-9516-113df0349e32 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Aug 28 00:42:06.946699 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:06.946699 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:06.946699 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:06.946699 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:06.946699 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:06.946699 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:06.946699 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:06.946699 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:06.946699 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:06.946699 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:06.946699 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:06.946699 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 28 00:42:06.946699 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 28 00:42:06.946699 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 28 00:42:06.946699 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 28 00:42:06.947723 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:06.947723 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:06.947723 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 28 00:42:06.947723 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 28 00:42:06.947723 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 28 00:42:06.947723 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 28 00:42:06.947723 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:06.947723 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:06.947723 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:06.947723 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:06.947723 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:06.947723 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:06.947723 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:06.947723 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:06.947723 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:06.947723 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:06.947723 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:06.947723 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:06.948688 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:06.948688 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:06.948688 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 28 00:42:06.948688 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 28 00:42:06.948688 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 28 00:42:06.948688 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 28 00:42:06.948688 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 28 00:42:06.948688 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 28 00:42:06.948688 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Aug 28 00:42:06.948688 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:06.948688 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 316/625] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Aug 28 00:42:06 2022] GET /identity/v3/users/glance => generated 84 bytes in 299 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:06.949824 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-474eb60d-6460-4cae-935a-1085ffd20f29 None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:06.949824 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-474eb60d-6460-4cae-935a-1085ffd20f29 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:06.950091 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-474eb60d-6460-4cae-935a-1085ffd20f29 None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:06.955927 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 317/626] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:42:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:42:06.956832 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-474eb60d-6460-4cae-935a-1085ffd20f29 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:07.008052 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2859fb9-7f2b-4520-8632-f958c02b556d None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:07.045620 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94bb842e-c47f-4f56-889a-a8060c3ee4fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:07.046483 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94bb842e-c47f-4f56-889a-a8060c3ee4fc None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:07.046763 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94bb842e-c47f-4f56-889a-a8060c3ee4fc None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:07.047041 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94bb842e-c47f-4f56-889a-a8060c3ee4fc None admin] PATH_INFO: `/v3/projects` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:07.047780 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94bb842e-c47f-4f56-889a-a8060c3ee4fc None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:07.048084 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94bb842e-c47f-4f56-889a-a8060c3ee4fc None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:07.049629 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94bb842e-c47f-4f56-889a-a8060c3ee4fc None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:07.063751 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 310/627] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Aug 28 00:42:06 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:07.065147 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbe94bbf-5662-4f21-b646-797dc1811867 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:07.065437 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbe94bbf-5662-4f21-b646-797dc1811867 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:07.065711 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbe94bbf-5662-4f21-b646-797dc1811867 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:07.084279 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-474eb60d-6460-4cae-935a-1085ffd20f29 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:07.086250 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-bbe94bbf-5662-4f21-b646-797dc1811867 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:42:07.087674 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-bbe94bbf-5662-4f21-b646-797dc1811867 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:42:07.109190 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:07.125528 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5555014-2c39-4e94-a9e7-13b2bebc4c54 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:07.126149 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5555014-2c39-4e94-a9e7-13b2bebc4c54 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:07.126434 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5555014-2c39-4e94-a9e7-13b2bebc4c54 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:07.126729 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5555014-2c39-4e94-a9e7-13b2bebc4c54 None admin] PATH_INFO: `/v3/users` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:07.127462 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5555014-2c39-4e94-a9e7-13b2bebc4c54 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:07.127768 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5555014-2c39-4e94-a9e7-13b2bebc4c54 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:07.129271 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5555014-2c39-4e94-a9e7-13b2bebc4c54 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:07.146753 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 318/628] 10.222.0.42 () {56 vars in 1137 bytes} [Sun Aug 28 00:42:06 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:07.147597 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5555014-2c39-4e94-a9e7-13b2bebc4c54 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:07.164816 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 311/629] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:42:07 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 28 00:42:07.166090 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28844267-32a1-4951-bf41-dbd143634347 None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:07.166379 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28844267-32a1-4951-bf41-dbd143634347 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:07.166655 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28844267-32a1-4951-bf41-dbd143634347 None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:07.167895 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 312/630] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:42:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:42:07.168904 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f525e45a-1db4-4a6a-9409-1be2490f61ed None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:07.169208 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f525e45a-1db4-4a6a-9409-1be2490f61ed None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:07.169481 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f525e45a-1db4-4a6a-9409-1be2490f61ed None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:07.171967 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 313/631] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:42:07 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:42:07.173017 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f525e45a-1db4-4a6a-9409-1be2490f61ed None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:07.197185 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:07.272224 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5555014-2c39-4e94-a9e7-13b2bebc4c54 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:07.298073 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f525e45a-1db4-4a6a-9409-1be2490f61ed None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:07.311133 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27e91a46-7c3c-427d-b61e-c00e6eb0c8ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:07.311800 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27e91a46-7c3c-427d-b61e-c00e6eb0c8ff None admin] REQUEST_METHOD: `PUT` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:07.312078 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27e91a46-7c3c-427d-b61e-c00e6eb0c8ff None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:07.312365 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27e91a46-7c3c-427d-b61e-c00e6eb0c8ff None admin] PATH_INFO: `/v3/projects/f53e6fdb369b480fbebb4086a505df67/users/ac4c5584582f4f55a7085af4cbbabf9b/roles/348070fea3dd4cc2a6e00e29fe82cc25` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:07.327457 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27e91a46-7c3c-427d-b61e-c00e6eb0c8ff None admin] RBAC: Authorizing `identity:create_grant(project_id=f53e6fdb369b480fbebb4086a505df67, user_id=ac4c5584582f4f55a7085af4cbbabf9b, role_id=348070fea3dd4cc2a6e00e29fe82cc25)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:07.328946 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27e91a46-7c3c-427d-b61e-c00e6eb0c8ff None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:07.337772 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a44ca531-986f-4bbd-ac5d-db99a3fd8f63 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:07.338417 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a44ca531-986f-4bbd-ac5d-db99a3fd8f63 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:07.338764 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a44ca531-986f-4bbd-ac5d-db99a3fd8f63 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:07.339032 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a44ca531-986f-4bbd-ac5d-db99a3fd8f63 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:07.352993 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 319/632] 10.222.0.42 () {58 vars in 1439 bytes} [Sun Aug 28 00:42:07 2022] PUT /identity/v3/projects/f53e6fdb369b480fbebb4086a505df67/users/ac4c5584582f4f55a7085af4cbbabf9b/roles/348070fea3dd4cc2a6e00e29fe82cc25 => generated 0 bytes in 198 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:42:07.352993 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dfa3ab5-c5af-4f56-b863-26d1f1ed5e8d None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:07.352993 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dfa3ab5-c5af-4f56-b863-26d1f1ed5e8d None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:07.352993 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dfa3ab5-c5af-4f56-b863-26d1f1ed5e8d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:07.363351 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:07.376182 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-8dfa3ab5-c5af-4f56-b863-26d1f1ed5e8d None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:42:07.377832 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-8dfa3ab5-c5af-4f56-b863-26d1f1ed5e8d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:42:07.418845 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a44ca531-986f-4bbd-ac5d-db99a3fd8f63 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:07.420503 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a44ca531-986f-4bbd-ac5d-db99a3fd8f63 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:07.472898 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 320/633] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:42:07 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 28 00:42:07.474496 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c150c09-bb80-49ee-89ea-acb64556fcfc None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:07.474787 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c150c09-bb80-49ee-89ea-acb64556fcfc None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:07.475062 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c150c09-bb80-49ee-89ea-acb64556fcfc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:07.495708 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-2c150c09-bb80-49ee-89ea-acb64556fcfc None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:42:07.497085 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-2c150c09-bb80-49ee-89ea-acb64556fcfc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:42:07.511269 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:07.556740 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 314/634] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Aug 28 00:42:07 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 384 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 28 00:42:07.557527 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a44ca531-986f-4bbd-ac5d-db99a3fd8f63 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:07.575825 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 321/635] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:42:07 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 28 00:42:07.576943 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c150c09-bb80-49ee-89ea-acb64556fcfc None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:07.629542 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:07.683215 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a44ca531-986f-4bbd-ac5d-db99a3fd8f63 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:07.689603 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c150c09-bb80-49ee-89ea-acb64556fcfc None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:07.721487 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81aa2367-73ae-48e1-b6b4-31401c81ce14 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:07.722133 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81aa2367-73ae-48e1-b6b4-31401c81ce14 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:07.722412 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81aa2367-73ae-48e1-b6b4-31401c81ce14 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:07.722685 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81aa2367-73ae-48e1-b6b4-31401c81ce14 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:07.726323 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7662aa0d-ddbd-4c4d-849a-2f71e2c9fc0a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:07.727236 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7662aa0d-ddbd-4c4d-849a-2f71e2c9fc0a None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:07.727709 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7662aa0d-ddbd-4c4d-849a-2f71e2c9fc0a None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:07.728227 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7662aa0d-ddbd-4c4d-849a-2f71e2c9fc0a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:07.734312 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81aa2367-73ae-48e1-b6b4-31401c81ce14 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 28 00:42:07.739504 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-81aa2367-73ae-48e1-b6b4-31401c81ce14 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 28 00:42:07.739504 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:07.739504 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:07.739504 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:07.739504 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:07.739504 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:07.739504 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:07.739504 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:07.739504 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:07.739504 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:07.739504 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:07.739504 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:07.739504 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 28 00:42:07.739504 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 28 00:42:07.739504 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 28 00:42:07.739504 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 28 00:42:07.740443 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 28 00:42:07.740443 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 28 00:42:07.740443 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 28 00:42:07.740443 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 28 00:42:07.740443 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:07.740443 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:07.740443 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:07.740443 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:07.740443 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:07.740443 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:07.740443 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:07.740443 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:07.740443 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:07.740443 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:07.740443 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:07.740443 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:07.741344 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:07.741344 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:07.741344 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:07.741344 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:07.741344 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 28 00:42:07.741344 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 28 00:42:07.741344 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 28 00:42:07.741344 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 28 00:42:07.741344 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 28 00:42:07.741344 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 28 00:42:07.741344 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 28 00:42:07.741344 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:42:07.741344 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 315/636] 10.222.0.42 () {56 vars in 1102 bytes} [Sun Aug 28 00:42:07 2022] GET /identity/v3/roles/service => generated 85 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:07.742841 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7662aa0d-ddbd-4c4d-849a-2f71e2c9fc0a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:07.747832 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7662aa0d-ddbd-4c4d-849a-2f71e2c9fc0a None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:07.748167 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81aa2367-73ae-48e1-b6b4-31401c81ce14 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:07.755973 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:07.767262 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-7662aa0d-ddbd-4c4d-849a-2f71e2c9fc0a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:42:07.767262 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:07.767262 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:07.767262 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:07.767262 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:07.767262 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:07.767262 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:07.767262 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:07.767262 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:07.767262 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:07.767262 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:07.767262 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:07.767262 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 28 00:42:07.767262 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 28 00:42:07.767262 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 28 00:42:07.767262 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 28 00:42:07.768267 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:07.768267 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:07.768267 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:07.768267 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:07.768267 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:07.768267 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:07.768267 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:07.768267 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:07.768267 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:07.768267 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:07.768267 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:07.768267 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:07.768267 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:07.768267 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:07.768267 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:07.768267 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:07.768267 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 28 00:42:07.769207 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 28 00:42:07.769207 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:42:07.769207 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:07.769207 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 322/637] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:42:07 2022] GET /identity/v3/domains/Default => generated 87 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:07.769969 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7662aa0d-ddbd-4c4d-849a-2f71e2c9fc0a None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:07.868854 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81aa2367-73ae-48e1-b6b4-31401c81ce14 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:07.882238 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7662aa0d-ddbd-4c4d-849a-2f71e2c9fc0a None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:07.884364 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:07.908649 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f83cadfd-4d12-4242-ba2d-2e09b5aceb0b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:07.909810 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f83cadfd-4d12-4242-ba2d-2e09b5aceb0b None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:07.910295 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f83cadfd-4d12-4242-ba2d-2e09b5aceb0b None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:07.910800 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f83cadfd-4d12-4242-ba2d-2e09b5aceb0b None admin] PATH_INFO: `/v3/roles/service` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:07.920357 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e624db94-9873-4a33-997c-162167fe5d36 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:07.921056 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e624db94-9873-4a33-997c-162167fe5d36 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:07.921337 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e624db94-9873-4a33-997c-162167fe5d36 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:07.921603 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e624db94-9873-4a33-997c-162167fe5d36 None admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:07.922272 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e624db94-9873-4a33-997c-162167fe5d36 None admin] RBAC: Adding query filter params (name=service) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:07.922582 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e624db94-9873-4a33-997c-162167fe5d36 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:07.924016 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e624db94-9873-4a33-997c-162167fe5d36 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:07.924720 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f83cadfd-4d12-4242-ba2d-2e09b5aceb0b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 28 00:42:07.930239 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-f83cadfd-4d12-4242-ba2d-2e09b5aceb0b None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 28 00:42:07.930239 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:07.930239 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:07.930239 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:07.930239 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:07.930239 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:07.930239 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:07.930239 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:07.930239 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:07.930239 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:07.930239 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:07.930239 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:07.930239 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 28 00:42:07.930239 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 28 00:42:07.930239 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 28 00:42:07.930239 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 28 00:42:07.931431 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 28 00:42:07.931431 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 28 00:42:07.931431 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 28 00:42:07.931431 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 28 00:42:07.931431 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:07.931431 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:07.931431 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:07.931431 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:07.931431 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:07.931431 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:07.931431 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:07.931431 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:07.931431 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:07.931431 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:07.931431 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:07.931431 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:07.932526 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:07.932526 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:07.932526 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:07.932526 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:07.932526 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 28 00:42:07.932526 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 28 00:42:07.932526 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 28 00:42:07.932526 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 28 00:42:07.932526 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 28 00:42:07.932526 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 28 00:42:07.932526 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 28 00:42:07.932526 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:42:07.933296 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 323/638] 10.222.0.42 () {56 vars in 1103 bytes} [Sun Aug 28 00:42:07 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:07.933601 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e624db94-9873-4a33-997c-162167fe5d36 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:07.935012 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 316/639] 10.222.0.42 () {56 vars in 1102 bytes} [Sun Aug 28 00:42:07 2022] GET /identity/v3/roles/service => generated 85 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:07.935996 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f83cadfd-4d12-4242-ba2d-2e09b5aceb0b None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:08.003877 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:08.044391 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f83cadfd-4d12-4242-ba2d-2e09b5aceb0b None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:08.050522 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e624db94-9873-4a33-997c-162167fe5d36 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:08.084333 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa45b75c-3099-4cd1-8d86-476989eb9acc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:08.085107 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa45b75c-3099-4cd1-8d86-476989eb9acc None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:08.085371 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa45b75c-3099-4cd1-8d86-476989eb9acc None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:08.085641 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa45b75c-3099-4cd1-8d86-476989eb9acc None admin] PATH_INFO: `/v3/roles/service` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:08.086780 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ba1b35e-96a3-4722-99fc-e5e000170446 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:08.087465 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ba1b35e-96a3-4722-99fc-e5e000170446 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:08.087672 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ba1b35e-96a3-4722-99fc-e5e000170446 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:08.087945 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ba1b35e-96a3-4722-99fc-e5e000170446 None admin] PATH_INFO: `/v3/domains` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:08.088831 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ba1b35e-96a3-4722-99fc-e5e000170446 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:08.088934 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ba1b35e-96a3-4722-99fc-e5e000170446 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:08.090308 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ba1b35e-96a3-4722-99fc-e5e000170446 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:08.097727 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa45b75c-3099-4cd1-8d86-476989eb9acc None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 28 00:42:08.102640 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 324/640] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Aug 28 00:42:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:08.102824 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-fa45b75c-3099-4cd1-8d86-476989eb9acc None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 28 00:42:08.102824 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:08.102824 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:08.102824 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:08.102824 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:08.102824 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:08.102824 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:08.102824 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:08.102824 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:08.102824 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:08.102824 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:08.102824 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:08.102824 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 28 00:42:08.102824 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 28 00:42:08.102824 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 28 00:42:08.102824 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 28 00:42:08.103998 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 28 00:42:08.103998 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 28 00:42:08.103998 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 28 00:42:08.103998 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 28 00:42:08.103998 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:08.103998 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:08.103998 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:08.103998 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:08.103998 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:08.103998 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:08.103998 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:08.103998 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:08.103998 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:08.103998 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:08.103998 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:08.103998 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:08.105074 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:08.105074 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:08.105074 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:08.105074 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:08.105074 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 28 00:42:08.105074 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 28 00:42:08.105074 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 28 00:42:08.105074 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 28 00:42:08.105074 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 28 00:42:08.105074 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 28 00:42:08.105074 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 28 00:42:08.105074 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:42:08.111305 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 317/641] 10.222.0.42 () {56 vars in 1102 bytes} [Sun Aug 28 00:42:07 2022] GET /identity/v3/roles/service => generated 85 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:08.111483 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ba1b35e-96a3-4722-99fc-e5e000170446 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:08.111596 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa45b75c-3099-4cd1-8d86-476989eb9acc None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:08.146933 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:08.217516 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa45b75c-3099-4cd1-8d86-476989eb9acc None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:08.233052 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ba1b35e-96a3-4722-99fc-e5e000170446 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:08.254070 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf4d7c3f-0284-45e5-89db-a7f6a92106f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:08.254699 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf4d7c3f-0284-45e5-89db-a7f6a92106f5 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:08.254984 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf4d7c3f-0284-45e5-89db-a7f6a92106f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:08.255269 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf4d7c3f-0284-45e5-89db-a7f6a92106f5 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:08.256108 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:08.266652 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf4d7c3f-0284-45e5-89db-a7f6a92106f5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:08.268124 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf4d7c3f-0284-45e5-89db-a7f6a92106f5 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:08.272193 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-076a9ec5-91dd-4326-a77f-8dacf3ada3c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:08.272807 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-076a9ec5-91dd-4326-a77f-8dacf3ada3c9 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:08.273120 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-076a9ec5-91dd-4326-a77f-8dacf3ada3c9 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:08.273393 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-076a9ec5-91dd-4326-a77f-8dacf3ada3c9 None admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:08.274056 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-076a9ec5-91dd-4326-a77f-8dacf3ada3c9 None admin] RBAC: Adding query filter params (name=service) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:08.274357 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-076a9ec5-91dd-4326-a77f-8dacf3ada3c9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:08.275716 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-076a9ec5-91dd-4326-a77f-8dacf3ada3c9 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:08.282653 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-bf4d7c3f-0284-45e5-89db-a7f6a92106f5 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:42:08.282653 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:08.282653 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:08.282653 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:08.282653 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:08.282653 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:08.282653 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:08.282653 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:08.282653 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:08.282653 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:08.282653 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:08.282653 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:08.282653 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 28 00:42:08.282653 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 28 00:42:08.282653 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 28 00:42:08.282653 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 28 00:42:08.283650 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:08.283650 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:08.283650 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:08.283650 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:08.283650 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:08.283650 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:08.283650 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:08.283650 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:08.283650 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:08.283650 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:08.283650 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:08.283650 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:08.283650 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:08.283650 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:08.283650 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:08.283650 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:08.283650 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 28 00:42:08.284569 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 28 00:42:08.284569 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:42:08.284569 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:42:08.284569 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 318/642] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:42:08 2022] GET /identity/v3/domains/Default => generated 87 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:08.285352 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf4d7c3f-0284-45e5-89db-a7f6a92106f5 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:08.289453 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 325/643] 10.222.0.42 () {56 vars in 1103 bytes} [Sun Aug 28 00:42:08 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:08.290190 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-076a9ec5-91dd-4326-a77f-8dacf3ada3c9 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:08.389614 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:08.398203 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-076a9ec5-91dd-4326-a77f-8dacf3ada3c9 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:08.406347 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf4d7c3f-0284-45e5-89db-a7f6a92106f5 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:08.434808 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d24000a-0ce3-4689-b5c9-8b878b0d9408 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:08.435598 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d24000a-0ce3-4689-b5c9-8b878b0d9408 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:08.435991 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d24000a-0ce3-4689-b5c9-8b878b0d9408 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:08.436287 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d24000a-0ce3-4689-b5c9-8b878b0d9408 None admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:08.437021 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d24000a-0ce3-4689-b5c9-8b878b0d9408 None admin] RBAC: Adding query filter params (name=service) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:08.437330 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d24000a-0ce3-4689-b5c9-8b878b0d9408 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:08.438709 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d24000a-0ce3-4689-b5c9-8b878b0d9408 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:08.443647 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24fe96d2-f1b2-4ece-8dfd-93bd6e2d7926 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:08.444513 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24fe96d2-f1b2-4ece-8dfd-93bd6e2d7926 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:08.444934 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24fe96d2-f1b2-4ece-8dfd-93bd6e2d7926 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:08.445379 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24fe96d2-f1b2-4ece-8dfd-93bd6e2d7926 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:08.447624 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 326/644] 10.222.0.42 () {56 vars in 1103 bytes} [Sun Aug 28 00:42:08 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:08.448651 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d24000a-0ce3-4689-b5c9-8b878b0d9408 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:08.457515 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24fe96d2-f1b2-4ece-8dfd-93bd6e2d7926 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:08.459173 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24fe96d2-f1b2-4ece-8dfd-93bd6e2d7926 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:08.471317 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-24fe96d2-f1b2-4ece-8dfd-93bd6e2d7926 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 28 00:42:08.471317 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:08.471317 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:08.471317 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:08.471317 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:08.471317 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:08.471317 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:08.471317 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:08.471317 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:08.471317 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:08.471317 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:08.471317 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:08.471317 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 28 00:42:08.471317 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 28 00:42:08.471317 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 28 00:42:08.472277 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 28 00:42:08.472277 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:08.472277 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:08.472277 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:08.472277 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:08.472277 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:08.472277 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:08.472277 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:08.472277 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:08.472277 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:08.472277 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:08.472277 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:08.472277 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:08.472277 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:08.472277 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:08.472277 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:08.472277 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:08.473647 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 28 00:42:08.473647 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 28 00:42:08.473647 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 28 00:42:08.473647 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 28 00:42:08.473647 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 28 00:42:08.473647 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 28 00:42:08.473647 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 28 00:42:08.473647 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:42:08.477121 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 319/645] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Aug 28 00:42:08 2022] GET /identity/v3/projects/service => generated 88 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:08.478162 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24fe96d2-f1b2-4ece-8dfd-93bd6e2d7926 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:08.502292 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:08.561905 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d24000a-0ce3-4689-b5c9-8b878b0d9408 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:08.585868 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24fe96d2-f1b2-4ece-8dfd-93bd6e2d7926 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:08.603954 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a68b9ff5-b634-4d85-88e5-92619cb843ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:08.604863 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a68b9ff5-b634-4d85-88e5-92619cb843ee None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:08.605172 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a68b9ff5-b634-4d85-88e5-92619cb843ee None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:08.605444 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a68b9ff5-b634-4d85-88e5-92619cb843ee None admin] PATH_INFO: `/v3/domains` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:08.606155 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a68b9ff5-b634-4d85-88e5-92619cb843ee None admin] RBAC: Adding query filter params (name=Default) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:08.606456 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a68b9ff5-b634-4d85-88e5-92619cb843ee None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:08.608350 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a68b9ff5-b634-4d85-88e5-92619cb843ee None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:08.621119 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c10540c-0e2c-4fc7-a259-dd223c48ba9b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:08.621842 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c10540c-0e2c-4fc7-a259-dd223c48ba9b None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:08.622407 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c10540c-0e2c-4fc7-a259-dd223c48ba9b None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:08.622518 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 327/646] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Aug 28 00:42:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:08.622624 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c10540c-0e2c-4fc7-a259-dd223c48ba9b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:08.627743 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a68b9ff5-b634-4d85-88e5-92619cb843ee None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:08.634023 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c10540c-0e2c-4fc7-a259-dd223c48ba9b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:08.635574 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c10540c-0e2c-4fc7-a259-dd223c48ba9b None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:08.638624 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:08.653232 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-8c10540c-0e2c-4fc7-a259-dd223c48ba9b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:42:08.653232 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:08.653232 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:08.653232 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:08.653232 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:08.653232 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:08.653232 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:08.653232 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:08.653232 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:08.653232 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:08.653232 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:08.653232 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:08.653232 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 28 00:42:08.653232 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 28 00:42:08.653232 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 28 00:42:08.653232 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 28 00:42:08.659192 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:08.659192 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:08.659192 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:08.659192 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:08.659192 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:08.659192 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:08.659192 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:08.659192 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:08.659192 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:08.659192 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:08.659192 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:08.659192 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:08.659192 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:08.659192 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:08.659192 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:08.659192 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:08.659192 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 28 00:42:08.660280 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 28 00:42:08.660280 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:42:08.660280 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:42:08.660280 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 320/647] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:42:08 2022] GET /identity/v3/domains/Default => generated 87 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:08.660280 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c10540c-0e2c-4fc7-a259-dd223c48ba9b None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:08.747735 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a68b9ff5-b634-4d85-88e5-92619cb843ee None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:08.750168 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:08.769031 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c10540c-0e2c-4fc7-a259-dd223c48ba9b None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:08.786823 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-733a6ee8-a346-4fa6-b077-e0f02ec9c071 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:08.787575 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-733a6ee8-a346-4fa6-b077-e0f02ec9c071 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:08.787922 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-733a6ee8-a346-4fa6-b077-e0f02ec9c071 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:08.788270 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-733a6ee8-a346-4fa6-b077-e0f02ec9c071 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:08.799933 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-733a6ee8-a346-4fa6-b077-e0f02ec9c071 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:08.801568 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-733a6ee8-a346-4fa6-b077-e0f02ec9c071 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:08.803489 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbabcd01-5799-46f5-9f3c-200c491f14b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:08.804177 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbabcd01-5799-46f5-9f3c-200c491f14b6 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:08.804462 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbabcd01-5799-46f5-9f3c-200c491f14b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:08.804740 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbabcd01-5799-46f5-9f3c-200c491f14b6 None admin] PATH_INFO: `/v3/projects` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:08.805462 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbabcd01-5799-46f5-9f3c-200c491f14b6 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:08.805767 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbabcd01-5799-46f5-9f3c-200c491f14b6 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:08.807209 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbabcd01-5799-46f5-9f3c-200c491f14b6 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:08.818789 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-733a6ee8-a346-4fa6-b077-e0f02ec9c071 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:42:08.818789 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:08.818789 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:08.818789 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:08.818789 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:08.818789 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:08.818789 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:08.818789 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:08.818789 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:08.818789 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:08.818789 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:08.818789 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:08.818789 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 28 00:42:08.818789 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 28 00:42:08.818789 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 28 00:42:08.818789 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 28 00:42:08.824952 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:08.824952 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:08.824952 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:08.824952 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:08.824952 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:08.824952 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:08.824952 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:08.824952 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:08.824952 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:08.824952 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:08.824952 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:08.824952 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:08.824952 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:08.824952 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:08.824952 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:08.824952 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:08.824952 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 28 00:42:08.826130 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 28 00:42:08.826130 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:42:08.826130 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:08.826338 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 321/648] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Aug 28 00:42:08 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:08.826424 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 328/649] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:42:08 2022] GET /identity/v3/domains/Default => generated 87 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:08.826424 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-733a6ee8-a346-4fa6-b077-e0f02ec9c071 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:08.826587 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbabcd01-5799-46f5-9f3c-200c491f14b6 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:08.882432 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:08.927008 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbabcd01-5799-46f5-9f3c-200c491f14b6 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:08.934067 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-733a6ee8-a346-4fa6-b077-e0f02ec9c071 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:08.962188 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24377fa6-2f69-4ab2-a711-0dd19a87f4fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:08.962896 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24377fa6-2f69-4ab2-a711-0dd19a87f4fa None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:08.963258 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24377fa6-2f69-4ab2-a711-0dd19a87f4fa None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:08.963592 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24377fa6-2f69-4ab2-a711-0dd19a87f4fa None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:08.970826 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b648f112-5beb-4600-81e0-fd13c2b5ca4a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:08.971286 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24377fa6-2f69-4ab2-a711-0dd19a87f4fa None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:08.971590 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b648f112-5beb-4600-81e0-fd13c2b5ca4a None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:08.971939 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b648f112-5beb-4600-81e0-fd13c2b5ca4a None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:08.972273 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b648f112-5beb-4600-81e0-fd13c2b5ca4a None admin] PATH_INFO: `/v3/domains` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:08.973009 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b648f112-5beb-4600-81e0-fd13c2b5ca4a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:08.973116 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24377fa6-2f69-4ab2-a711-0dd19a87f4fa None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:08.973437 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b648f112-5beb-4600-81e0-fd13c2b5ca4a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:08.975118 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b648f112-5beb-4600-81e0-fd13c2b5ca4a None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:08.985382 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-24377fa6-2f69-4ab2-a711-0dd19a87f4fa None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Aug 28 00:42:08.985382 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:08.985382 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:08.985382 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:08.985382 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:08.985382 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:08.985382 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:08.985382 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:08.985382 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:08.985382 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:08.985382 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:08.985382 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:08.985382 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 28 00:42:08.985382 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 28 00:42:08.985382 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 28 00:42:08.985382 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 28 00:42:08.986557 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:08.986557 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:08.986557 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 28 00:42:08.986557 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 28 00:42:08.986557 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 28 00:42:08.986557 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 28 00:42:08.986557 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:08.986557 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:08.986557 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:08.986557 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:08.986557 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:08.986557 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:08.986557 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:08.986557 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:08.986557 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:08.986557 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:08.986557 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:08.986557 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:08.987707 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:08.987707 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:08.987707 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 28 00:42:08.987707 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 28 00:42:08.987707 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 28 00:42:08.987707 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 28 00:42:08.987707 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 28 00:42:08.987707 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 28 00:42:08.987707 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Aug 28 00:42:08.987707 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:42:08.994753 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 329/650] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Aug 28 00:42:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:08.994753 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b648f112-5beb-4600-81e0-fd13c2b5ca4a None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:08.994996 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 322/651] 10.222.0.42 () {56 vars in 1102 bytes} [Sun Aug 28 00:42:08 2022] GET /identity/v3/users/neutron => generated 85 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:08.994996 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24377fa6-2f69-4ab2-a711-0dd19a87f4fa None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:09.012248 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:09.097645 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b648f112-5beb-4600-81e0-fd13c2b5ca4a None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:09.098069 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24377fa6-2f69-4ab2-a711-0dd19a87f4fa None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:09.118713 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:09.141530 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b932a0e3-14d6-4c69-b5a1-5ed4effbedc3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:09.142284 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b932a0e3-14d6-4c69-b5a1-5ed4effbedc3 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:09.142494 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-561922fc-12a4-42ac-9490-820c893516e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:09.142610 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b932a0e3-14d6-4c69-b5a1-5ed4effbedc3 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:09.142824 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b932a0e3-14d6-4c69-b5a1-5ed4effbedc3 None admin] PATH_INFO: `/v3/domains` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:09.143074 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-561922fc-12a4-42ac-9490-820c893516e6 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:09.143351 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-561922fc-12a4-42ac-9490-820c893516e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:09.143466 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b932a0e3-14d6-4c69-b5a1-5ed4effbedc3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:09.143647 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-561922fc-12a4-42ac-9490-820c893516e6 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:09.143755 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b932a0e3-14d6-4c69-b5a1-5ed4effbedc3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:09.144391 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-561922fc-12a4-42ac-9490-820c893516e6 None admin] RBAC: Adding query filter params (role.id=348070fea3dd4cc2a6e00e29fe82cc25, scope.project.id=f53e6fdb369b480fbebb4086a505df67, user.id=14176b54cf6640abb3b1c14e3824d4f1) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:09.144697 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-561922fc-12a4-42ac-9490-820c893516e6 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:09.145275 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b932a0e3-14d6-4c69-b5a1-5ed4effbedc3 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:09.146169 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-561922fc-12a4-42ac-9490-820c893516e6 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:09.155082 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 330/652] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Aug 28 00:42:08 2022] GET /identity/v3/role_assignments?user.id=14176b54cf6640abb3b1c14e3824d4f1&scope.project.id=f53e6fdb369b480fbebb4086a505df67&role.id=348070fea3dd4cc2a6e00e29fe82cc25 => generated 608 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:09.156203 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-561922fc-12a4-42ac-9490-820c893516e6 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:09.167948 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 323/653] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Aug 28 00:42:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:09.169371 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b932a0e3-14d6-4c69-b5a1-5ed4effbedc3 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:09.254805 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:09.277273 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-561922fc-12a4-42ac-9490-820c893516e6 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:09.302199 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b932a0e3-14d6-4c69-b5a1-5ed4effbedc3 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:09.328343 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e86c5627-b717-4c22-a640-abda88237195 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:09.329464 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e86c5627-b717-4c22-a640-abda88237195 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:09.330443 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e86c5627-b717-4c22-a640-abda88237195 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:09.330946 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e86c5627-b717-4c22-a640-abda88237195 None admin] PATH_INFO: `/v3/users` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:09.331773 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e86c5627-b717-4c22-a640-abda88237195 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:09.332736 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e86c5627-b717-4c22-a640-abda88237195 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:09.334554 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e86c5627-b717-4c22-a640-abda88237195 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:09.356247 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d052eb26-31a5-4f92-8588-6ec80f7fa8c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:09.356922 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 331/654] 10.222.0.42 () {56 vars in 1139 bytes} [Sun Aug 28 00:42:09 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:09.357087 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d052eb26-31a5-4f92-8588-6ec80f7fa8c5 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:09.357166 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d052eb26-31a5-4f92-8588-6ec80f7fa8c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:09.357436 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d052eb26-31a5-4f92-8588-6ec80f7fa8c5 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:09.358626 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e86c5627-b717-4c22-a640-abda88237195 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:09.367118 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d052eb26-31a5-4f92-8588-6ec80f7fa8c5 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:09.369216 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d052eb26-31a5-4f92-8588-6ec80f7fa8c5 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:09.384145 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-d052eb26-31a5-4f92-8588-6ec80f7fa8c5 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Aug 28 00:42:09.384145 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:09.384145 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:09.384145 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:09.384145 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:09.384145 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:09.384145 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:09.384145 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:09.384145 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:09.384145 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:09.384145 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:09.384145 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:09.384145 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 28 00:42:09.384145 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 28 00:42:09.384145 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 28 00:42:09.384145 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 28 00:42:09.385175 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:09.385175 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:09.385175 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 28 00:42:09.385175 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 28 00:42:09.385175 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 28 00:42:09.385175 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 28 00:42:09.385175 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:09.385175 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:09.385175 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:09.385175 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:09.385175 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:09.385175 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:09.385175 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:09.385175 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:09.385175 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:09.385175 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:09.385175 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:09.385175 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:09.386139 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:09.386139 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:09.386139 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 28 00:42:09.386139 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 28 00:42:09.386139 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 28 00:42:09.386139 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 28 00:42:09.386139 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 28 00:42:09.386139 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 28 00:42:09.386139 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Aug 28 00:42:09.386139 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:42:09.386139 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 324/655] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Aug 28 00:42:09 2022] GET /identity/v3/users/cinder => generated 84 bytes in 217 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:09.386733 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d052eb26-31a5-4f92-8588-6ec80f7fa8c5 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:09.429840 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:09.488524 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e86c5627-b717-4c22-a640-abda88237195 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:09.498661 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d052eb26-31a5-4f92-8588-6ec80f7fa8c5 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:09.533997 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03e5693e-690b-460e-8a92-af0678f36dba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:09.534195 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07a4e92a-1e0c-41e9-ad4a-0d9334fa3e54 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:09.534525 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07a4e92a-1e0c-41e9-ad4a-0d9334fa3e54 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:09.534630 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03e5693e-690b-460e-8a92-af0678f36dba None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:09.534798 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07a4e92a-1e0c-41e9-ad4a-0d9334fa3e54 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:09.534896 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03e5693e-690b-460e-8a92-af0678f36dba None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:09.535066 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03e5693e-690b-460e-8a92-af0678f36dba None admin] PATH_INFO: `/v3/users/nova` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:09.535162 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07a4e92a-1e0c-41e9-ad4a-0d9334fa3e54 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:09.543096 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03e5693e-690b-460e-8a92-af0678f36dba None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:09.544582 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03e5693e-690b-460e-8a92-af0678f36dba None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:09.546959 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:09.559486 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-03e5693e-690b-460e-8a92-af0678f36dba None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 28 00:42:09.559486 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:09.559486 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:09.559486 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:09.559486 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:09.559486 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:09.559486 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:09.559486 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:09.559486 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:09.559486 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:09.559486 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:09.559486 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:09.559486 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 28 00:42:09.559486 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 28 00:42:09.559486 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 28 00:42:09.559486 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 28 00:42:09.560716 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:09.560716 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:09.560716 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 28 00:42:09.560716 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 28 00:42:09.560716 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 28 00:42:09.560716 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 28 00:42:09.560716 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:09.560716 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:09.560716 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:09.560716 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:09.560716 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:09.560716 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:09.560716 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:09.560716 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:09.560716 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:09.560716 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:09.560716 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:09.560716 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:09.561852 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:09.561852 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:09.561852 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 28 00:42:09.561852 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 28 00:42:09.561852 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 28 00:42:09.561852 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 28 00:42:09.561852 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 28 00:42:09.561852 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 28 00:42:09.561852 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 28 00:42:09.561852 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:09.565019 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 332/656] 10.222.0.42 () {56 vars in 1093 bytes} [Sun Aug 28 00:42:09 2022] GET /identity/v3/users/nova => generated 82 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:09.565896 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03e5693e-690b-460e-8a92-af0678f36dba None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:09.612761 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07a4e92a-1e0c-41e9-ad4a-0d9334fa3e54 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:09.614400 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07a4e92a-1e0c-41e9-ad4a-0d9334fa3e54 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:09.663683 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:09.688199 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03e5693e-690b-460e-8a92-af0678f36dba None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:09.729095 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19e4e132-5f76-4a54-8164-ec7dc6f064d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:09.729707 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19e4e132-5f76-4a54-8164-ec7dc6f064d2 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:09.729985 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19e4e132-5f76-4a54-8164-ec7dc6f064d2 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:09.730264 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19e4e132-5f76-4a54-8164-ec7dc6f064d2 None admin] PATH_INFO: `/v3/users` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:09.730999 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19e4e132-5f76-4a54-8164-ec7dc6f064d2 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:09.731300 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19e4e132-5f76-4a54-8164-ec7dc6f064d2 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:09.733416 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19e4e132-5f76-4a54-8164-ec7dc6f064d2 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:09.748809 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 325/657] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Aug 28 00:42:09 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 363 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 28 00:42:09.750280 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08e2feb0-db75-419c-8186-5aacf126858c None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:09.750566 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08e2feb0-db75-419c-8186-5aacf126858c None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:09.750835 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08e2feb0-db75-419c-8186-5aacf126858c None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:09.752097 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 326/658] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:42:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:42:09.752791 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08e2feb0-db75-419c-8186-5aacf126858c None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:09.761384 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 333/659] 10.222.0.42 () {56 vars in 1137 bytes} [Sun Aug 28 00:42:09 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:09.762252 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19e4e132-5f76-4a54-8164-ec7dc6f064d2 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:09.824612 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:09.876389 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19e4e132-5f76-4a54-8164-ec7dc6f064d2 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:09.877250 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08e2feb0-db75-419c-8186-5aacf126858c None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:09.913380 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8af8038-4542-48a0-aec6-10a69e36cc3c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:09.913630 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a4b5d7b-d5cc-4921-946c-03495b3745c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:09.914030 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a4b5d7b-d5cc-4921-946c-03495b3745c0 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:09.914311 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a4b5d7b-d5cc-4921-946c-03495b3745c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:09.914578 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a4b5d7b-d5cc-4921-946c-03495b3745c0 None admin] PATH_INFO: `/v3/users` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:09.915143 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8af8038-4542-48a0-aec6-10a69e36cc3c None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:09.915325 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a4b5d7b-d5cc-4921-946c-03495b3745c0 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:09.915597 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a4b5d7b-d5cc-4921-946c-03495b3745c0 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:09.915987 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8af8038-4542-48a0-aec6-10a69e36cc3c None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:09.916450 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8af8038-4542-48a0-aec6-10a69e36cc3c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:09.917089 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a4b5d7b-d5cc-4921-946c-03495b3745c0 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:09.919919 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:09.930154 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8af8038-4542-48a0-aec6-10a69e36cc3c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:09.931869 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8af8038-4542-48a0-aec6-10a69e36cc3c None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:09.936240 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 327/660] 10.222.0.42 () {56 vars in 1133 bytes} [Sun Aug 28 00:42:09 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:09.937671 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99c5494c-e4d4-4820-90d7-a2b39b8f30c7 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:09.937957 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99c5494c-e4d4-4820-90d7-a2b39b8f30c7 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:09.938225 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99c5494c-e4d4-4820-90d7-a2b39b8f30c7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:09.949699 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-f8af8038-4542-48a0-aec6-10a69e36cc3c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:42:09.949699 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:09.949699 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:09.949699 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:09.949699 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:09.949699 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:09.949699 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:09.949699 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:09.949699 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:09.949699 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:09.949699 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:09.949699 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:09.949699 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 28 00:42:09.949699 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 28 00:42:09.949699 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 28 00:42:09.949699 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 28 00:42:09.950884 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:09.950884 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:09.950884 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:09.950884 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:09.950884 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:09.950884 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:09.950884 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:09.950884 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:09.950884 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:09.950884 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:09.950884 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:09.950884 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:09.950884 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:09.950884 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:09.950884 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:09.950884 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:09.950884 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 28 00:42:09.951835 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 28 00:42:09.951835 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:42:09.951835 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:09.952746 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 334/661] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:42:09 2022] GET /identity/v3/domains/Default => generated 87 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:09.953672 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8af8038-4542-48a0-aec6-10a69e36cc3c None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:09.959834 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-99c5494c-e4d4-4820-90d7-a2b39b8f30c7 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:42:09.961106 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-99c5494c-e4d4-4820-90d7-a2b39b8f30c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:42:10.029788 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 328/662] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:42:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 28 00:42:10.031246 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99c5494c-e4d4-4820-90d7-a2b39b8f30c7 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:10.075381 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8af8038-4542-48a0-aec6-10a69e36cc3c None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:10.089776 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:10.125049 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02549472-705d-4221-8b56-79a23d3b5f1f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:10.125962 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02549472-705d-4221-8b56-79a23d3b5f1f None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:10.126527 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02549472-705d-4221-8b56-79a23d3b5f1f None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:10.126996 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02549472-705d-4221-8b56-79a23d3b5f1f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:10.139025 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99c5494c-e4d4-4820-90d7-a2b39b8f30c7 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:10.164952 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:10.177625 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2faff05-3070-4094-ab27-34751ec48baf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:10.178281 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2faff05-3070-4094-ab27-34751ec48baf None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:10.178618 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2faff05-3070-4094-ab27-34751ec48baf None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:10.178886 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2faff05-3070-4094-ab27-34751ec48baf None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:10.205936 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02549472-705d-4221-8b56-79a23d3b5f1f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:10.207481 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02549472-705d-4221-8b56-79a23d3b5f1f None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:10.258933 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2faff05-3070-4094-ab27-34751ec48baf None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:10.261260 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2faff05-3070-4094-ab27-34751ec48baf None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:10.335872 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 335/663] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Aug 28 00:42:09 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 382 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 28 00:42:10.336740 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02549472-705d-4221-8b56-79a23d3b5f1f None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:10.347948 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:10.390361 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 329/664] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Aug 28 00:42:10 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 360 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 28 00:42:10.391861 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2faff05-3070-4094-ab27-34751ec48baf None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:10.419020 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:10.458228 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02549472-705d-4221-8b56-79a23d3b5f1f None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:10.500170 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e50542ed-0d64-4f1d-ad76-d431e0dfb0b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:10.500815 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e50542ed-0d64-4f1d-ad76-d431e0dfb0b4 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:10.501075 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e50542ed-0d64-4f1d-ad76-d431e0dfb0b4 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:10.501348 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e50542ed-0d64-4f1d-ad76-d431e0dfb0b4 None admin] PATH_INFO: `/v3/domains` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:10.501989 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e50542ed-0d64-4f1d-ad76-d431e0dfb0b4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:10.502289 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e50542ed-0d64-4f1d-ad76-d431e0dfb0b4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:10.503175 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2faff05-3070-4094-ab27-34751ec48baf None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:10.503663 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e50542ed-0d64-4f1d-ad76-d431e0dfb0b4 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:10.519912 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 336/665] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Aug 28 00:42:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:10.519912 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e50542ed-0d64-4f1d-ad76-d431e0dfb0b4 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:10.544974 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-346ee9b5-3df4-49c9-adce-868af2f55ce5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:10.545799 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-346ee9b5-3df4-49c9-adce-868af2f55ce5 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:10.546084 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-346ee9b5-3df4-49c9-adce-868af2f55ce5 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:10.546348 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-346ee9b5-3df4-49c9-adce-868af2f55ce5 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:10.557533 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-346ee9b5-3df4-49c9-adce-868af2f55ce5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 28 00:42:10.562477 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-346ee9b5-3df4-49c9-adce-868af2f55ce5 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 28 00:42:10.562477 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:10.562477 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:10.562477 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:10.562477 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:10.562477 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:10.562477 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:10.562477 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:10.562477 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:10.562477 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:10.562477 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:10.562477 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:10.562477 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 28 00:42:10.562477 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 28 00:42:10.562477 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 28 00:42:10.562477 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 28 00:42:10.563408 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 28 00:42:10.563408 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 28 00:42:10.563408 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 28 00:42:10.563408 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 28 00:42:10.563408 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:10.563408 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:10.563408 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:10.563408 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:10.563408 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:10.563408 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:10.563408 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:10.563408 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:10.563408 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:10.563408 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:10.563408 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:10.563408 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:10.564239 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:10.564239 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:10.564239 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:10.564239 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:10.564239 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 28 00:42:10.564239 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 28 00:42:10.564239 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 28 00:42:10.564239 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 28 00:42:10.564239 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 28 00:42:10.564239 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 28 00:42:10.564239 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 28 00:42:10.564239 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:42:10.568419 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 330/666] 10.222.0.42 () {56 vars in 1102 bytes} [Sun Aug 28 00:42:10 2022] GET /identity/v3/roles/service => generated 85 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:10.569264 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-346ee9b5-3df4-49c9-adce-868af2f55ce5 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:10.593045 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:10.628920 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e50542ed-0d64-4f1d-ad76-d431e0dfb0b4 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:10.655988 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:10.665968 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33fc2c50-d787-463e-a6c4-37e2387dcf4a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:10.666601 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33fc2c50-d787-463e-a6c4-37e2387dcf4a None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:10.666881 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33fc2c50-d787-463e-a6c4-37e2387dcf4a None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:10.667154 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33fc2c50-d787-463e-a6c4-37e2387dcf4a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:10.672789 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-346ee9b5-3df4-49c9-adce-868af2f55ce5 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:10.678473 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33fc2c50-d787-463e-a6c4-37e2387dcf4a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:10.679943 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33fc2c50-d787-463e-a6c4-37e2387dcf4a None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:10.694986 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-33fc2c50-d787-463e-a6c4-37e2387dcf4a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:42:10.694986 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:10.694986 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:10.694986 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:10.694986 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:10.694986 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:10.694986 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:10.694986 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:10.694986 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:10.694986 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:10.694986 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:10.694986 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:10.694986 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 28 00:42:10.694986 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 28 00:42:10.694986 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 28 00:42:10.694986 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 28 00:42:10.695955 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:10.695955 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:10.695955 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:10.695955 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:10.695955 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:10.695955 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:10.695955 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:10.695955 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:10.695955 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:10.695955 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:10.695955 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:10.695955 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:10.695955 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:10.695955 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:10.695955 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:10.695955 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:10.695955 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 28 00:42:10.696833 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 28 00:42:10.696833 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:42:10.696833 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:10.700830 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 337/667] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:42:10 2022] GET /identity/v3/domains/Default => generated 87 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:10.701561 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33fc2c50-d787-463e-a6c4-37e2387dcf4a None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:10.712013 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c3302ef-a717-4894-a571-93e42aee9b97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:10.712650 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c3302ef-a717-4894-a571-93e42aee9b97 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:10.712925 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c3302ef-a717-4894-a571-93e42aee9b97 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:10.713225 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c3302ef-a717-4894-a571-93e42aee9b97 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:10.724430 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c3302ef-a717-4894-a571-93e42aee9b97 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:10.725930 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c3302ef-a717-4894-a571-93e42aee9b97 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:10.741579 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-8c3302ef-a717-4894-a571-93e42aee9b97 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:42:10.741579 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:10.741579 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:10.741579 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:10.741579 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:10.741579 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:10.741579 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:10.741579 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:10.741579 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:10.741579 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:10.741579 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:10.741579 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:10.741579 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 28 00:42:10.741579 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 28 00:42:10.741579 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 28 00:42:10.741579 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 28 00:42:10.742752 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:10.742752 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:10.742752 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:10.742752 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:10.742752 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:10.742752 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:10.742752 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:10.742752 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:10.742752 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:10.742752 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:10.742752 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:10.742752 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:10.742752 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:10.742752 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:10.742752 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:10.742752 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:10.742752 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 28 00:42:10.743617 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 28 00:42:10.743617 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:42:10.743617 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:42:10.743805 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 331/668] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:42:10 2022] GET /identity/v3/domains/Default => generated 87 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:10.745087 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c3302ef-a717-4894-a571-93e42aee9b97 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:10.817294 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33fc2c50-d787-463e-a6c4-37e2387dcf4a None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:10.839073 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:10.854756 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a71d0aa7-78af-452e-89f0-488a6c062629 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:10.855744 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a71d0aa7-78af-452e-89f0-488a6c062629 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:10.856018 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a71d0aa7-78af-452e-89f0-488a6c062629 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:10.856290 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a71d0aa7-78af-452e-89f0-488a6c062629 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:10.858438 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c3302ef-a717-4894-a571-93e42aee9b97 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:10.863601 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a71d0aa7-78af-452e-89f0-488a6c062629 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:10.865112 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a71d0aa7-78af-452e-89f0-488a6c062629 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:10.877313 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-a71d0aa7-78af-452e-89f0-488a6c062629 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 28 00:42:10.877313 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:10.877313 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:10.877313 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:10.877313 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:10.877313 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:10.877313 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:10.877313 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:10.877313 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:10.877313 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:10.877313 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:10.877313 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:10.877313 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 28 00:42:10.877313 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 28 00:42:10.877313 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 28 00:42:10.878233 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 28 00:42:10.878233 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:10.878233 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:10.878233 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:10.878233 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:10.878233 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:10.878233 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:10.878233 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:10.878233 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:10.878233 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:10.878233 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:10.878233 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:10.878233 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:10.878233 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:10.878233 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:10.878233 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:10.878233 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:10.883972 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 28 00:42:10.883972 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 28 00:42:10.883972 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 28 00:42:10.883972 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 28 00:42:10.883972 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 28 00:42:10.883972 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 28 00:42:10.883972 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 28 00:42:10.883972 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:10.883972 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 338/669] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Aug 28 00:42:10 2022] GET /identity/v3/projects/service => generated 88 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:10.883972 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a71d0aa7-78af-452e-89f0-488a6c062629 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:10.899787 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-477ee4c3-a9d2-4cd9-8ee5-1e2d2e64749c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:10.900399 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-477ee4c3-a9d2-4cd9-8ee5-1e2d2e64749c None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:10.900678 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-477ee4c3-a9d2-4cd9-8ee5-1e2d2e64749c None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:10.900948 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-477ee4c3-a9d2-4cd9-8ee5-1e2d2e64749c None admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:10.901634 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-477ee4c3-a9d2-4cd9-8ee5-1e2d2e64749c None admin] RBAC: Adding query filter params (name=service) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:10.901930 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-477ee4c3-a9d2-4cd9-8ee5-1e2d2e64749c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:10.903311 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-477ee4c3-a9d2-4cd9-8ee5-1e2d2e64749c None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:10.906174 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:10.918680 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 332/670] 10.222.0.42 () {56 vars in 1103 bytes} [Sun Aug 28 00:42:10 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:10.918680 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-477ee4c3-a9d2-4cd9-8ee5-1e2d2e64749c None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:10.996100 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a71d0aa7-78af-452e-89f0-488a6c062629 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:11.032813 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-477ee4c3-a9d2-4cd9-8ee5-1e2d2e64749c None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:11.035377 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-930a2d09-d423-4e5a-b9ec-ad646713a3da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:11.035988 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-930a2d09-d423-4e5a-b9ec-ad646713a3da None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:11.036268 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-930a2d09-d423-4e5a-b9ec-ad646713a3da None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:11.036540 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-930a2d09-d423-4e5a-b9ec-ad646713a3da None admin] PATH_INFO: `/v3/domains` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:11.037209 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-930a2d09-d423-4e5a-b9ec-ad646713a3da None admin] RBAC: Adding query filter params (name=Default) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:11.037511 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-930a2d09-d423-4e5a-b9ec-ad646713a3da None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:11.038892 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-930a2d09-d423-4e5a-b9ec-ad646713a3da None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:11.051910 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 339/671] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Aug 28 00:42:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:11.053123 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-930a2d09-d423-4e5a-b9ec-ad646713a3da None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:11.072391 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cb8a7f9-007d-4d85-953f-5c7b51f725c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:11.073047 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cb8a7f9-007d-4d85-953f-5c7b51f725c1 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:11.073331 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cb8a7f9-007d-4d85-953f-5c7b51f725c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:11.073608 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cb8a7f9-007d-4d85-953f-5c7b51f725c1 None admin] PATH_INFO: `/v3/domains` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:11.074274 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cb8a7f9-007d-4d85-953f-5c7b51f725c1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:11.074576 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cb8a7f9-007d-4d85-953f-5c7b51f725c1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:11.076004 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cb8a7f9-007d-4d85-953f-5c7b51f725c1 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:11.088718 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 333/672] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Aug 28 00:42:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:11.089610 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cb8a7f9-007d-4d85-953f-5c7b51f725c1 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:11.108575 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:11.157758 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:11.175550 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-930a2d09-d423-4e5a-b9ec-ad646713a3da None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:11.202225 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cb8a7f9-007d-4d85-953f-5c7b51f725c1 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:11.213859 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccf27ce8-3a14-461b-b4d9-2eb917823f00 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:11.214477 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccf27ce8-3a14-461b-b4d9-2eb917823f00 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:11.214761 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccf27ce8-3a14-461b-b4d9-2eb917823f00 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:11.215046 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccf27ce8-3a14-461b-b4d9-2eb917823f00 None admin] PATH_INFO: `/v3/projects` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:11.215744 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccf27ce8-3a14-461b-b4d9-2eb917823f00 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:11.216056 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccf27ce8-3a14-461b-b4d9-2eb917823f00 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:11.217548 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccf27ce8-3a14-461b-b4d9-2eb917823f00 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:11.233967 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 340/673] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Aug 28 00:42:11 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:11.233967 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccf27ce8-3a14-461b-b4d9-2eb917823f00 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:11.239685 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44951d30-948c-4cee-b438-01be5c55a3cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:11.240291 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44951d30-948c-4cee-b438-01be5c55a3cc None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:11.240568 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44951d30-948c-4cee-b438-01be5c55a3cc None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:11.240839 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44951d30-948c-4cee-b438-01be5c55a3cc None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:11.251597 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44951d30-948c-4cee-b438-01be5c55a3cc None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:11.253097 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44951d30-948c-4cee-b438-01be5c55a3cc None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:11.267454 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-44951d30-948c-4cee-b438-01be5c55a3cc None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:42:11.267454 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:11.267454 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:11.267454 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:11.267454 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:11.267454 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:11.267454 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:11.267454 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:11.267454 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:11.267454 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:11.267454 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:11.267454 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:11.267454 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 28 00:42:11.267454 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 28 00:42:11.267454 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 28 00:42:11.267454 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 28 00:42:11.268610 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:11.268610 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:11.268610 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:11.268610 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:11.268610 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:11.268610 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:11.268610 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:11.268610 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:11.268610 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:11.268610 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:11.268610 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:11.268610 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:11.268610 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:11.268610 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:11.268610 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:11.268610 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:11.268610 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 28 00:42:11.269747 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 28 00:42:11.269747 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:42:11.269747 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:42:11.269747 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 334/674] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:42:11 2022] GET /identity/v3/domains/Default => generated 87 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:11.269747 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44951d30-948c-4cee-b438-01be5c55a3cc None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:11.335102 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:11.338240 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccf27ce8-3a14-461b-b4d9-2eb917823f00 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:11.369274 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44951d30-948c-4cee-b438-01be5c55a3cc None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:11.372653 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24bf114e-c0ae-4b9c-b3f7-3cbf76ea0a0e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:11.373327 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24bf114e-c0ae-4b9c-b3f7-3cbf76ea0a0e None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:11.373611 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24bf114e-c0ae-4b9c-b3f7-3cbf76ea0a0e None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:11.373915 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24bf114e-c0ae-4b9c-b3f7-3cbf76ea0a0e None admin] PATH_INFO: `/v3/projects/service` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:11.379459 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:11.380548 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24bf114e-c0ae-4b9c-b3f7-3cbf76ea0a0e None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:11.382245 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24bf114e-c0ae-4b9c-b3f7-3cbf76ea0a0e None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:11.393867 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-24bf114e-c0ae-4b9c-b3f7-3cbf76ea0a0e None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 28 00:42:11.393867 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:11.393867 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:11.393867 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:11.393867 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:11.393867 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:11.393867 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:11.393867 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:11.393867 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:11.393867 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:11.393867 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:11.393867 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:11.393867 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 28 00:42:11.393867 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 28 00:42:11.393867 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 28 00:42:11.394932 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 28 00:42:11.394932 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:11.394932 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:11.394932 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:11.394932 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:11.394932 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:11.394932 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:11.394932 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:11.394932 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:11.394932 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:11.394932 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:11.394932 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:11.394932 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:11.394932 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:11.394932 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:11.394932 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:11.394932 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:11.395998 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 28 00:42:11.395998 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 28 00:42:11.395998 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 28 00:42:11.395998 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 28 00:42:11.395998 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 28 00:42:11.395998 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 28 00:42:11.395998 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 28 00:42:11.395998 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:11.399860 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 341/675] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Aug 28 00:42:11 2022] GET /identity/v3/projects/service => generated 88 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:11.400899 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b989b27-53c5-4d4f-845d-9fac21130cfa None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:11.401211 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b989b27-53c5-4d4f-845d-9fac21130cfa None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:11.401486 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b989b27-53c5-4d4f-845d-9fac21130cfa None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:11.402742 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 342/676] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:42:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:42:11.403429 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b989b27-53c5-4d4f-845d-9fac21130cfa None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:11.405898 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24f9650f-b4d3-440d-847b-6c76b6b97c96 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:11.406534 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f9650f-b4d3-440d-847b-6c76b6b97c96 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:11.406928 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f9650f-b4d3-440d-847b-6c76b6b97c96 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:11.407205 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f9650f-b4d3-440d-847b-6c76b6b97c96 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:11.414225 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24f9650f-b4d3-440d-847b-6c76b6b97c96 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:11.415699 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24f9650f-b4d3-440d-847b-6c76b6b97c96 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:11.427951 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-24f9650f-b4d3-440d-847b-6c76b6b97c96 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 28 00:42:11.427951 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:11.427951 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:11.427951 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:11.427951 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:11.427951 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:11.427951 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:11.427951 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:11.427951 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:11.427951 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:11.427951 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:11.427951 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:11.427951 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 28 00:42:11.427951 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 28 00:42:11.427951 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 28 00:42:11.428878 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 28 00:42:11.428878 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:11.428878 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:11.428878 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:11.428878 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:11.428878 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:11.428878 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:11.428878 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:11.428878 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:11.428878 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:11.428878 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:11.428878 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:11.428878 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:11.428878 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:11.428878 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:11.428878 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:11.428878 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:11.429822 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 28 00:42:11.429822 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 28 00:42:11.429822 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 28 00:42:11.429822 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 28 00:42:11.429822 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 28 00:42:11.429822 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 28 00:42:11.429822 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 28 00:42:11.429822 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:42:11.429822 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 335/677] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Aug 28 00:42:11 2022] GET /identity/v3/projects/service => generated 88 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:11.429822 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24f9650f-b4d3-440d-847b-6c76b6b97c96 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:11.513233 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b989b27-53c5-4d4f-845d-9fac21130cfa None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:11.531285 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24f9650f-b4d3-440d-847b-6c76b6b97c96 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:11.549000 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67fdc76c-9bd6-42e4-8a54-02ed3dd1e659 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:11.549637 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67fdc76c-9bd6-42e4-8a54-02ed3dd1e659 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:11.549887 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67fdc76c-9bd6-42e4-8a54-02ed3dd1e659 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:11.550173 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67fdc76c-9bd6-42e4-8a54-02ed3dd1e659 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:11.550882 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67fdc76c-9bd6-42e4-8a54-02ed3dd1e659 None admin] RBAC: Adding query filter params (role.id=348070fea3dd4cc2a6e00e29fe82cc25, scope.project.id=f53e6fdb369b480fbebb4086a505df67, user.id=364a28c108e744728917ba2b0aa2347d) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:11.551184 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67fdc76c-9bd6-42e4-8a54-02ed3dd1e659 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:11.552604 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67fdc76c-9bd6-42e4-8a54-02ed3dd1e659 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:11.558924 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 343/678] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Aug 28 00:42:11 2022] GET /identity/v3/role_assignments?user.id=364a28c108e744728917ba2b0aa2347d&scope.project.id=f53e6fdb369b480fbebb4086a505df67&role.id=348070fea3dd4cc2a6e00e29fe82cc25 => generated 608 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:11.559558 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67fdc76c-9bd6-42e4-8a54-02ed3dd1e659 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:11.579591 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e30e8b6a-a7a7-43be-b5cf-87224f4935bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:11.580590 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e30e8b6a-a7a7-43be-b5cf-87224f4935bc None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:11.580865 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e30e8b6a-a7a7-43be-b5cf-87224f4935bc None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:11.581204 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e30e8b6a-a7a7-43be-b5cf-87224f4935bc None admin] PATH_INFO: `/v3/domains` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:11.581871 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e30e8b6a-a7a7-43be-b5cf-87224f4935bc None admin] RBAC: Adding query filter params (name=Default) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:11.582170 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e30e8b6a-a7a7-43be-b5cf-87224f4935bc None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:11.583636 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e30e8b6a-a7a7-43be-b5cf-87224f4935bc None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:11.589146 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:11.598750 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 336/679] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Aug 28 00:42:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:11.599862 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-550431d9-0632-48b3-8388-f19b7191db13 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:11.600145 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-550431d9-0632-48b3-8388-f19b7191db13 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:11.600471 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-550431d9-0632-48b3-8388-f19b7191db13 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:11.620271 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-550431d9-0632-48b3-8388-f19b7191db13 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:42:11.621481 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-550431d9-0632-48b3-8388-f19b7191db13 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:42:11.647874 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:11.676193 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67fdc76c-9bd6-42e4-8a54-02ed3dd1e659 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:11.699897 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 337/680] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:42:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 28 00:42:11.700778 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-550431d9-0632-48b3-8388-f19b7191db13 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:11.725333 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1388ac7-4a16-41c1-81cf-509f0cd8c725 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:11.726635 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1388ac7-4a16-41c1-81cf-509f0cd8c725 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:11.726922 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1388ac7-4a16-41c1-81cf-509f0cd8c725 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:11.727198 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1388ac7-4a16-41c1-81cf-509f0cd8c725 None admin] PATH_INFO: `/v3/projects` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:11.727909 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1388ac7-4a16-41c1-81cf-509f0cd8c725 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:11.728221 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1388ac7-4a16-41c1-81cf-509f0cd8c725 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:11.729731 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1388ac7-4a16-41c1-81cf-509f0cd8c725 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:11.746802 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 344/681] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Aug 28 00:42:11 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:11.747273 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1388ac7-4a16-41c1-81cf-509f0cd8c725 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:11.831845 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-550431d9-0632-48b3-8388-f19b7191db13 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:11.848559 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:11.868634 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1388ac7-4a16-41c1-81cf-509f0cd8c725 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:11.872028 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01481955-723c-4bdd-a45a-76b4805c57a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:11.872423 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01481955-723c-4bdd-a45a-76b4805c57a1 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:11.872699 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01481955-723c-4bdd-a45a-76b4805c57a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:11.872983 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01481955-723c-4bdd-a45a-76b4805c57a1 None admin] PATH_INFO: `/v3/projects` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:11.873695 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01481955-723c-4bdd-a45a-76b4805c57a1 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:11.874030 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01481955-723c-4bdd-a45a-76b4805c57a1 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:11.875495 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01481955-723c-4bdd-a45a-76b4805c57a1 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:11.893033 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 338/682] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Aug 28 00:42:11 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:11.893033 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01481955-723c-4bdd-a45a-76b4805c57a1 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:11.912740 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:11.915476 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3fe84ef-aa60-4f7e-adfc-3e9f1a30269a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:11.916460 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3fe84ef-aa60-4f7e-adfc-3e9f1a30269a None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:11.916946 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3fe84ef-aa60-4f7e-adfc-3e9f1a30269a None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:11.917456 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3fe84ef-aa60-4f7e-adfc-3e9f1a30269a None admin] PATH_INFO: `/v3/users/swift` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:11.927371 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3fe84ef-aa60-4f7e-adfc-3e9f1a30269a None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:11.929160 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3fe84ef-aa60-4f7e-adfc-3e9f1a30269a None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:11.943219 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-b3fe84ef-aa60-4f7e-adfc-3e9f1a30269a None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Aug 28 00:42:11.943219 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:11.943219 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:11.943219 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:11.943219 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:11.943219 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:11.943219 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:11.943219 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:11.943219 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:11.943219 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:11.943219 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:11.943219 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:11.943219 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 28 00:42:11.943219 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 28 00:42:11.943219 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 28 00:42:11.943219 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 28 00:42:11.944482 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:11.944482 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:11.944482 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 28 00:42:11.944482 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 28 00:42:11.944482 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 28 00:42:11.944482 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 28 00:42:11.944482 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:11.944482 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:11.944482 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:11.944482 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:11.944482 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:11.944482 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:11.944482 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:11.944482 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:11.944482 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:11.944482 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:11.944482 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:11.944482 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:11.945656 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:11.945656 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:11.945656 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 28 00:42:11.945656 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 28 00:42:11.945656 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 28 00:42:11.945656 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 28 00:42:11.945656 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 28 00:42:11.945656 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 28 00:42:11.945656 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Aug 28 00:42:11.945656 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:11.947958 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 345/683] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Aug 28 00:42:11 2022] GET /identity/v3/users/swift => generated 83 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:11.949394 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3fe84ef-aa60-4f7e-adfc-3e9f1a30269a None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:11.995619 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01481955-723c-4bdd-a45a-76b4805c57a1 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:12.028373 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8286f1e3-c9e7-457b-9243-7ba6f3201d35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:12.028988 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8286f1e3-c9e7-457b-9243-7ba6f3201d35 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:12.029276 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8286f1e3-c9e7-457b-9243-7ba6f3201d35 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:12.029544 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8286f1e3-c9e7-457b-9243-7ba6f3201d35 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:12.030320 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8286f1e3-c9e7-457b-9243-7ba6f3201d35 None admin] RBAC: Adding query filter params (role.id=348070fea3dd4cc2a6e00e29fe82cc25, scope.project.id=f53e6fdb369b480fbebb4086a505df67, user.id=879afc99ebb9402b94fadc6091040c5c) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:12.030622 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8286f1e3-c9e7-457b-9243-7ba6f3201d35 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:12.032062 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8286f1e3-c9e7-457b-9243-7ba6f3201d35 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:12.038268 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 339/684] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Aug 28 00:42:11 2022] GET /identity/v3/role_assignments?user.id=879afc99ebb9402b94fadc6091040c5c&scope.project.id=f53e6fdb369b480fbebb4086a505df67&role.id=348070fea3dd4cc2a6e00e29fe82cc25 => generated 608 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:12.039079 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8286f1e3-c9e7-457b-9243-7ba6f3201d35 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:12.079502 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3fe84ef-aa60-4f7e-adfc-3e9f1a30269a None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:12.128097 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fd755cb-8e2f-4197-9103-1ca4f00e21c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:12.128734 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fd755cb-8e2f-4197-9103-1ca4f00e21c8 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:12.129069 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fd755cb-8e2f-4197-9103-1ca4f00e21c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:12.129348 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fd755cb-8e2f-4197-9103-1ca4f00e21c8 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:12.130151 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fd755cb-8e2f-4197-9103-1ca4f00e21c8 None admin] RBAC: Adding query filter params (role.id=348070fea3dd4cc2a6e00e29fe82cc25, scope.project.id=f53e6fdb369b480fbebb4086a505df67, user.id=168b07df50c24593ba73dbde5b037aed) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:12.130650 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fd755cb-8e2f-4197-9103-1ca4f00e21c8 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:12.132196 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fd755cb-8e2f-4197-9103-1ca4f00e21c8 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:12.145417 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 346/685] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Aug 28 00:42:11 2022] GET /identity/v3/role_assignments?user.id=168b07df50c24593ba73dbde5b037aed&scope.project.id=f53e6fdb369b480fbebb4086a505df67&role.id=348070fea3dd4cc2a6e00e29fe82cc25 => generated 608 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:12.146435 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fd755cb-8e2f-4197-9103-1ca4f00e21c8 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:12.170774 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:12.185901 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:12.222911 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8286f1e3-c9e7-457b-9243-7ba6f3201d35 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:12.264641 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65ef2cf5-b11c-49c3-8632-819e1f94a1ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:12.264641 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ef2cf5-b11c-49c3-8632-819e1f94a1ea None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:12.265178 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ef2cf5-b11c-49c3-8632-819e1f94a1ea None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:12.265519 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ef2cf5-b11c-49c3-8632-819e1f94a1ea None admin] PATH_INFO: `/v3/users` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:12.266342 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65ef2cf5-b11c-49c3-8632-819e1f94a1ea None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:12.266713 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65ef2cf5-b11c-49c3-8632-819e1f94a1ea None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:12.268304 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65ef2cf5-b11c-49c3-8632-819e1f94a1ea None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:12.291866 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 340/686] 10.222.0.42 () {56 vars in 1135 bytes} [Sun Aug 28 00:42:12 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 253 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:12.298128 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65ef2cf5-b11c-49c3-8632-819e1f94a1ea None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:12.300782 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fd755cb-8e2f-4197-9103-1ca4f00e21c8 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:12.343711 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b392d4e9-a46a-4e88-8741-86725e0442a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:12.344603 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b392d4e9-a46a-4e88-8741-86725e0442a0 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:12.345050 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b392d4e9-a46a-4e88-8741-86725e0442a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:12.345457 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b392d4e9-a46a-4e88-8741-86725e0442a0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:12.358292 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b392d4e9-a46a-4e88-8741-86725e0442a0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:12.360083 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b392d4e9-a46a-4e88-8741-86725e0442a0 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:12.376813 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-b392d4e9-a46a-4e88-8741-86725e0442a0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:42:12.376813 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:12.376813 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:12.376813 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:12.376813 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:12.376813 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:12.376813 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:12.376813 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:12.376813 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:12.376813 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:12.376813 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:12.376813 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:12.376813 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 28 00:42:12.376813 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 28 00:42:12.376813 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 28 00:42:12.376813 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 28 00:42:12.377987 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:12.377987 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:12.377987 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:12.377987 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:12.377987 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:12.377987 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:12.377987 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:12.377987 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:12.377987 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:12.377987 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:12.377987 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:12.377987 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:12.377987 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:12.377987 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:12.377987 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:12.377987 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:12.377987 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 28 00:42:12.378993 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 28 00:42:12.378993 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:42:12.378993 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:12.380829 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 347/687] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:42:12 2022] GET /identity/v3/domains/Default => generated 87 bytes in 235 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:12.386675 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b392d4e9-a46a-4e88-8741-86725e0442a0 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:12.419484 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65ef2cf5-b11c-49c3-8632-819e1f94a1ea None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:12.458416 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:12.465677 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecf3c539-a03c-4f71-905f-574547ee8ba8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:12.466305 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf3c539-a03c-4f71-905f-574547ee8ba8 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:12.466587 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf3c539-a03c-4f71-905f-574547ee8ba8 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:12.466863 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf3c539-a03c-4f71-905f-574547ee8ba8 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:12.484014 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:12.507933 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b392d4e9-a46a-4e88-8741-86725e0442a0 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:12.555465 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecf3c539-a03c-4f71-905f-574547ee8ba8 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:12.557085 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecf3c539-a03c-4f71-905f-574547ee8ba8 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:12.558110 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dde12b6a-1083-4d3c-b190-649521e073c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:12.558799 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dde12b6a-1083-4d3c-b190-649521e073c5 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:12.559089 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dde12b6a-1083-4d3c-b190-649521e073c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:12.559358 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dde12b6a-1083-4d3c-b190-649521e073c5 None admin] PATH_INFO: `/v3/domains` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:12.560072 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dde12b6a-1083-4d3c-b190-649521e073c5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:12.560376 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dde12b6a-1083-4d3c-b190-649521e073c5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:12.562537 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dde12b6a-1083-4d3c-b190-649521e073c5 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:12.580871 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 348/688] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Aug 28 00:42:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 28 00:42:12.586878 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dde12b6a-1083-4d3c-b190-649521e073c5 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:12.667682 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 341/689] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Aug 28 00:42:12 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 370 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Aug 28 00:42:12.678011 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecf3c539-a03c-4f71-905f-574547ee8ba8 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:12.712610 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dde12b6a-1083-4d3c-b190-649521e073c5 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:12.735201 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:12.755401 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e55cae90-d562-4ce4-8e26-8f342d20a61b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:12.756029 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e55cae90-d562-4ce4-8e26-8f342d20a61b None admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:12.756309 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e55cae90-d562-4ce4-8e26-8f342d20a61b None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:12.756507 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:12.756611 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e55cae90-d562-4ce4-8e26-8f342d20a61b None admin] PATH_INFO: `/v3/users` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:12.757770 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e55cae90-d562-4ce4-8e26-8f342d20a61b None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:12.759247 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e55cae90-d562-4ce4-8e26-8f342d20a61b None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:12.803405 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 349/690] 10.222.0.42 () {60 vars in 1132 bytes} [Sun Aug 28 00:42:12 2022] POST /identity/v3/users => generated 254 bytes in 217 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:42:12.846558 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecf3c539-a03c-4f71-905f-574547ee8ba8 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:12.888988 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c53c0a09-5af2-45be-b650-cff18dc62fb0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:12.888988 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c53c0a09-5af2-45be-b650-cff18dc62fb0 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:12.888988 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c53c0a09-5af2-45be-b650-cff18dc62fb0 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:12.888988 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c53c0a09-5af2-45be-b650-cff18dc62fb0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:12.900984 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c53c0a09-5af2-45be-b650-cff18dc62fb0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:12.902264 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c53c0a09-5af2-45be-b650-cff18dc62fb0 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:12.917854 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-c53c0a09-5af2-45be-b650-cff18dc62fb0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:42:12.917854 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:12.917854 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:12.917854 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:12.917854 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:12.917854 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:12.917854 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:12.917854 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:12.917854 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:12.917854 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:12.917854 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:12.917854 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:12.917854 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 28 00:42:12.917854 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 28 00:42:12.917854 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 28 00:42:12.917854 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 28 00:42:12.918934 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:12.918934 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:12.918934 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:12.918934 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:12.918934 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:12.918934 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:12.918934 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:12.918934 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:12.918934 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:12.918934 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:12.918934 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:12.918934 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:12.918934 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:12.918934 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:12.918934 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:12.918934 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:12.918934 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 28 00:42:12.919806 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 28 00:42:12.919806 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:42:12.919806 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:42:12.919806 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 342/691] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:42:12 2022] GET /identity/v3/domains/Default => generated 87 bytes in 242 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:12.931468 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e55cae90-d562-4ce4-8e26-8f342d20a61b None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:13.059102 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e55cae90-d562-4ce4-8e26-8f342d20a61b None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:13.116513 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d14db50-b0f7-4416-a580-01eb64143c64 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:13.125061 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d14db50-b0f7-4416-a580-01eb64143c64 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:13.125061 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d14db50-b0f7-4416-a580-01eb64143c64 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:13.125061 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d14db50-b0f7-4416-a580-01eb64143c64 None admin] PATH_INFO: `/v3/domains` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:13.125061 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d14db50-b0f7-4416-a580-01eb64143c64 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:13.125061 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d14db50-b0f7-4416-a580-01eb64143c64 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:13.125061 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d14db50-b0f7-4416-a580-01eb64143c64 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:13.148607 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 350/692] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Aug 28 00:42:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:13.154090 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c53c0a09-5af2-45be-b650-cff18dc62fb0 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:13.285237 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:13.290372 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c53c0a09-5af2-45be-b650-cff18dc62fb0 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:13.327319 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83109a66-095c-4a6e-b01f-8a07cae8fadc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:13.327971 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83109a66-095c-4a6e-b01f-8a07cae8fadc None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:13.328255 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83109a66-095c-4a6e-b01f-8a07cae8fadc None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:13.328990 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83109a66-095c-4a6e-b01f-8a07cae8fadc None admin] PATH_INFO: `/v3/projects/service` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:13.336373 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83109a66-095c-4a6e-b01f-8a07cae8fadc None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:13.337934 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83109a66-095c-4a6e-b01f-8a07cae8fadc None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:13.350826 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-83109a66-095c-4a6e-b01f-8a07cae8fadc None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 28 00:42:13.350826 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:13.350826 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:13.350826 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:13.350826 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:13.350826 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:13.350826 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:13.350826 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:13.350826 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:13.350826 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:13.350826 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:13.350826 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:13.350826 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 28 00:42:13.350826 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 28 00:42:13.350826 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 28 00:42:13.351761 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 28 00:42:13.351761 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:13.351761 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:13.351761 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:13.351761 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:13.351761 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:13.351761 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:13.351761 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:13.351761 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:13.351761 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:13.351761 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:13.351761 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:13.351761 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:13.351761 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:13.351761 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:13.351761 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:13.351761 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:13.352643 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 28 00:42:13.352643 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 28 00:42:13.352643 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 28 00:42:13.352643 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 28 00:42:13.352643 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 28 00:42:13.352643 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 28 00:42:13.352643 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 28 00:42:13.352643 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:42:13.352643 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 343/693] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Aug 28 00:42:13 2022] GET /identity/v3/projects/service => generated 88 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:13.358430 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d14db50-b0f7-4416-a580-01eb64143c64 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:13.445371 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:13.523808 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d14db50-b0f7-4416-a580-01eb64143c64 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:13.569154 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc322ac1-c08f-47db-b196-9be5192cb5b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:13.569791 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc322ac1-c08f-47db-b196-9be5192cb5b7 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:13.570027 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc322ac1-c08f-47db-b196-9be5192cb5b7 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:13.570222 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc322ac1-c08f-47db-b196-9be5192cb5b7 None admin] PATH_INFO: `/v3/projects` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:13.571076 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc322ac1-c08f-47db-b196-9be5192cb5b7 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:13.571384 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc322ac1-c08f-47db-b196-9be5192cb5b7 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:13.572818 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc322ac1-c08f-47db-b196-9be5192cb5b7 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:13.588278 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 351/694] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Aug 28 00:42:13 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 230 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:13.594753 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83109a66-095c-4a6e-b01f-8a07cae8fadc None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:13.705592 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83109a66-095c-4a6e-b01f-8a07cae8fadc None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:13.740156 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-470f9dd8-da5e-45e4-983b-55ea75dc9239 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:13.741424 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-470f9dd8-da5e-45e4-983b-55ea75dc9239 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:13.741912 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-470f9dd8-da5e-45e4-983b-55ea75dc9239 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:13.742379 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-470f9dd8-da5e-45e4-983b-55ea75dc9239 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:13.743324 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-470f9dd8-da5e-45e4-983b-55ea75dc9239 None admin] RBAC: Adding query filter params (role.id=348070fea3dd4cc2a6e00e29fe82cc25, scope.project.id=f53e6fdb369b480fbebb4086a505df67, user.id=ac4c5584582f4f55a7085af4cbbabf9b) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:13.743847 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-470f9dd8-da5e-45e4-983b-55ea75dc9239 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:13.745521 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-470f9dd8-da5e-45e4-983b-55ea75dc9239 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:13.752584 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 344/695] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Aug 28 00:42:13 2022] GET /identity/v3/role_assignments?user.id=ac4c5584582f4f55a7085af4cbbabf9b&scope.project.id=f53e6fdb369b480fbebb4086a505df67&role.id=348070fea3dd4cc2a6e00e29fe82cc25 => generated 608 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:13.899443 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-427d0932-2b3c-446e-9629-b713c3f1df4e None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:13.899838 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-427d0932-2b3c-446e-9629-b713c3f1df4e None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:13.900044 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-427d0932-2b3c-446e-9629-b713c3f1df4e None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:13.901807 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 352/696] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:42:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:42:13.907674 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16afec2e-fc23-463b-af59-d006ef707f7c None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:13.907974 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16afec2e-fc23-463b-af59-d006ef707f7c None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:13.908231 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16afec2e-fc23-463b-af59-d006ef707f7c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:13.927598 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-16afec2e-fc23-463b-af59-d006ef707f7c None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:42:13.928825 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-16afec2e-fc23-463b-af59-d006ef707f7c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:42:13.955065 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:14.014639 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 345/697] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:42:13 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 28 00:42:14.294939 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-427d0932-2b3c-446e-9629-b713c3f1df4e None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:14.399579 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:14.410750 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-427d0932-2b3c-446e-9629-b713c3f1df4e None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:14.457174 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2de06a5-85f7-4509-b06f-81cb6e94715a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:14.457410 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa6dae1-d1e4-4fe6-85c1-d58ee0b29ea5 None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:14.457628 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa6dae1-d1e4-4fe6-85c1-d58ee0b29ea5 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:14.457893 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa6dae1-d1e4-4fe6-85c1-d58ee0b29ea5 None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:14.458690 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2de06a5-85f7-4509-b06f-81cb6e94715a None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:14.459183 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2de06a5-85f7-4509-b06f-81cb6e94715a None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:14.459562 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 346/698] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:42:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 28 00:42:14.459839 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2de06a5-85f7-4509-b06f-81cb6e94715a None admin] PATH_INFO: `/v3/services/network` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:14.460609 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2de06a5-85f7-4509-b06f-81cb6e94715a None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:14.462212 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2de06a5-85f7-4509-b06f-81cb6e94715a None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:14.464786 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cce6e40d-bfc0-4ac8-b341-9e33756867c1 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:14.465322 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cce6e40d-bfc0-4ac8-b341-9e33756867c1 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:14.465597 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cce6e40d-bfc0-4ac8-b341-9e33756867c1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:14.474410 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-b2de06a5-85f7-4509-b06f-81cb6e94715a None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Aug 28 00:42:14.474410 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:14.474410 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 28 00:42:14.474410 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 28 00:42:14.474410 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 28 00:42:14.474410 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 28 00:42:14.474410 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 28 00:42:14.474410 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 28 00:42:14.474410 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Aug 28 00:42:14.474410 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:14.474410 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 28 00:42:14.474410 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:14.474410 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:14.474410 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:14.474410 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:14.474410 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:14.474410 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:14.475462 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:14.475462 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:14.475462 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:14.475462 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:14.475462 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:14.475462 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:14.475462 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 28 00:42:14.475462 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 28 00:42:14.475462 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 28 00:42:14.475462 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 28 00:42:14.475462 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:14.475462 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:14.475462 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:14.475462 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:14.475462 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:14.475462 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:14.476310 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:14.476310 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:14.476310 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:14.476310 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:14.476310 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:14.476310 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:14.476310 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:14.476310 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:14.476310 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:14.476310 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:14.476310 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 28 00:42:14.476310 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 28 00:42:14.476310 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Aug 28 00:42:14.476310 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:14.476310 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 353/699] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Aug 28 00:42:14 2022] GET /identity/v3/services/network => generated 88 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:14.481694 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2de06a5-85f7-4509-b06f-81cb6e94715a None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:14.502251 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-cce6e40d-bfc0-4ac8-b341-9e33756867c1 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:42:14.503477 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-cce6e40d-bfc0-4ac8-b341-9e33756867c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:42:14.633876 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 347/700] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:42:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 28 00:42:14.673416 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2de06a5-85f7-4509-b06f-81cb6e94715a None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:14.709801 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b15e424-e22d-44fa-b4e0-b8b8d2fa3d23 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:14.710419 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b15e424-e22d-44fa-b4e0-b8b8d2fa3d23 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:14.710696 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b15e424-e22d-44fa-b4e0-b8b8d2fa3d23 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:14.710968 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b15e424-e22d-44fa-b4e0-b8b8d2fa3d23 None admin] PATH_INFO: `/v3/services` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:14.711601 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b15e424-e22d-44fa-b4e0-b8b8d2fa3d23 None admin] RBAC: Adding query filter params (name=network) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:14.711899 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b15e424-e22d-44fa-b4e0-b8b8d2fa3d23 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:14.713388 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b15e424-e22d-44fa-b4e0-b8b8d2fa3d23 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:14.718612 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 354/701] 10.222.0.42 () {56 vars in 1112 bytes} [Sun Aug 28 00:42:14 2022] GET /identity/v3/services?name=network => generated 124 bytes in 237 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:14.723895 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cce6e40d-bfc0-4ac8-b341-9e33756867c1 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:14.830601 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:14.856911 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cce6e40d-bfc0-4ac8-b341-9e33756867c1 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:14.903623 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05ab8d2b-77ae-4947-ad31-184a220f144c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:14.904156 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05ab8d2b-77ae-4947-ad31-184a220f144c None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:14.904433 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05ab8d2b-77ae-4947-ad31-184a220f144c None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:14.904702 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05ab8d2b-77ae-4947-ad31-184a220f144c None admin] PATH_INFO: `/v3/services` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:14.905548 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05ab8d2b-77ae-4947-ad31-184a220f144c None admin] RBAC: Adding query filter params (type=network) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:14.905823 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05ab8d2b-77ae-4947-ad31-184a220f144c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:14.907206 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05ab8d2b-77ae-4947-ad31-184a220f144c None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:14.913424 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 348/702] 10.222.0.42 () {56 vars in 1112 bytes} [Sun Aug 28 00:42:14 2022] GET /identity/v3/services?type=network => generated 124 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:14.920142 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f97464e4-4f2c-4452-9011-bc0f468daac9 None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:14.920846 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f97464e4-4f2c-4452-9011-bc0f468daac9 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:14.921366 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f97464e4-4f2c-4452-9011-bc0f468daac9 None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:14.922993 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 355/703] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:42:14 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:42:14.936155 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d7bb106-5e61-4cd4-9c1e-484a23ee516a None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:14.936540 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d7bb106-5e61-4cd4-9c1e-484a23ee516a None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:14.936759 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d7bb106-5e61-4cd4-9c1e-484a23ee516a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:14.966262 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-3d7bb106-5e61-4cd4-9c1e-484a23ee516a None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:42:14.967592 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-3d7bb106-5e61-4cd4-9c1e-484a23ee516a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:42:14.998654 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f97464e4-4f2c-4452-9011-bc0f468daac9 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:15.041488 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:15.046842 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 349/704] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:42:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 28 00:42:15.128507 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f97464e4-4f2c-4452-9011-bc0f468daac9 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:15.164332 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4acac4d4-4650-45bb-a34f-a572f2eef2f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:15.164902 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4acac4d4-4650-45bb-a34f-a572f2eef2f3 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:15.165226 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4acac4d4-4650-45bb-a34f-a572f2eef2f3 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:15.165502 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4acac4d4-4650-45bb-a34f-a572f2eef2f3 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:15.166096 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4acac4d4-4650-45bb-a34f-a572f2eef2f3 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:15.167475 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4acac4d4-4650-45bb-a34f-a572f2eef2f3 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:15.176851 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-4acac4d4-4650-45bb-a34f-a572f2eef2f3 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 28 00:42:15.176851 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:15.176851 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 28 00:42:15.176851 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 28 00:42:15.176851 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 28 00:42:15.176851 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 28 00:42:15.176851 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 28 00:42:15.176851 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 28 00:42:15.176851 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 28 00:42:15.176851 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:15.176851 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 28 00:42:15.176851 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:15.176851 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:15.176851 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:15.176851 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:15.176851 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:15.176851 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:15.177880 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:15.177880 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:15.177880 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:15.177880 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:15.177880 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:15.177880 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:15.177880 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 28 00:42:15.177880 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 28 00:42:15.177880 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 28 00:42:15.177880 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 28 00:42:15.177880 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:15.177880 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:15.177880 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:15.177880 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:15.177880 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:15.177880 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:15.178731 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:15.178731 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:15.178731 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:15.178731 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:15.178731 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:15.178731 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:15.178731 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:15.178731 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:15.178731 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:15.178731 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:15.178731 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 28 00:42:15.178731 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 28 00:42:15.178731 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 28 00:42:15.178731 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:15.178731 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 356/705] 10.222.0.42 () {56 vars in 1129 bytes} [Sun Aug 28 00:42:14 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:15.183963 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d7bb106-5e61-4cd4-9c1e-484a23ee516a None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:15.260561 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:15.305557 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d7bb106-5e61-4cd4-9c1e-484a23ee516a None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:15.350173 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-708408bd-ba99-4ad3-a9db-f73040ca9344 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:15.350857 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-708408bd-ba99-4ad3-a9db-f73040ca9344 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:15.351139 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-708408bd-ba99-4ad3-a9db-f73040ca9344 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:15.351414 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-708408bd-ba99-4ad3-a9db-f73040ca9344 None admin] PATH_INFO: `/v3/services` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:15.352060 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-708408bd-ba99-4ad3-a9db-f73040ca9344 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:15.352360 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-708408bd-ba99-4ad3-a9db-f73040ca9344 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:15.353866 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-708408bd-ba99-4ad3-a9db-f73040ca9344 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:15.364366 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 350/706] 10.222.0.42 () {56 vars in 1124 bytes} [Sun Aug 28 00:42:15 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 28 00:42:15.370122 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4acac4d4-4650-45bb-a34f-a572f2eef2f3 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:15.385069 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-708408bd-ba99-4ad3-a9db-f73040ca9344 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:15.535436 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:15.550213 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4acac4d4-4650-45bb-a34f-a572f2eef2f3 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:15.579127 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-708408bd-ba99-4ad3-a9db-f73040ca9344 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:15.590307 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-183bfa5b-785c-4d1c-8c09-24f464f56435 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:15.591216 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-183bfa5b-785c-4d1c-8c09-24f464f56435 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:15.591720 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-183bfa5b-785c-4d1c-8c09-24f464f56435 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:15.592208 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-183bfa5b-785c-4d1c-8c09-24f464f56435 None admin] PATH_INFO: `/v3/services` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:15.593193 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-183bfa5b-785c-4d1c-8c09-24f464f56435 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:15.593751 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-183bfa5b-785c-4d1c-8c09-24f464f56435 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:15.595330 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-183bfa5b-785c-4d1c-8c09-24f464f56435 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:15.601899 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 357/707] 10.222.0.42 () {56 vars in 1124 bytes} [Sun Aug 28 00:42:15 2022] GET /identity/v3/services?type=block-storage => generated 130 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:15.638607 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0bc6caf-d1a7-43db-855f-dfb8b6ea8990 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:15.640053 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0bc6caf-d1a7-43db-855f-dfb8b6ea8990 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:15.640578 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0bc6caf-d1a7-43db-855f-dfb8b6ea8990 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:15.641168 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0bc6caf-d1a7-43db-855f-dfb8b6ea8990 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:15.650033 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b55e39c5-eb47-46ad-94c4-7073abf1b546 None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:15.650309 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b55e39c5-eb47-46ad-94c4-7073abf1b546 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:15.650780 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b55e39c5-eb47-46ad-94c4-7073abf1b546 None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:15.652517 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 358/708] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:42:15 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 28 00:42:15.653673 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:15.654765 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0bc6caf-d1a7-43db-855f-dfb8b6ea8990 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 28 00:42:15.659615 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ff891b7-3813-413d-8a50-57152469278c None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:15.660169 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-b0bc6caf-d1a7-43db-855f-dfb8b6ea8990 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 28 00:42:15.660169 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:15.660169 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:15.660169 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:15.660169 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:15.660169 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:15.660169 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:15.660169 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:15.660169 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:15.660169 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:15.660169 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:15.660169 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:15.660169 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 28 00:42:15.660169 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 28 00:42:15.660169 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 28 00:42:15.660169 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 28 00:42:15.661304 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 28 00:42:15.661304 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 28 00:42:15.661304 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 28 00:42:15.661304 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 28 00:42:15.661304 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:15.661304 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:15.661304 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:15.661304 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:15.661304 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:15.661304 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:15.661304 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:15.661304 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:15.661304 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:15.661304 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:15.661304 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:15.661304 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:15.662134 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:15.662134 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:15.662134 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:15.662134 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:15.662134 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 28 00:42:15.662134 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 28 00:42:15.662134 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 28 00:42:15.662134 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 28 00:42:15.662134 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 28 00:42:15.662134 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 28 00:42:15.662134 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 28 00:42:15.662134 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:42:15.663850 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ff891b7-3813-413d-8a50-57152469278c None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:15.664286 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ff891b7-3813-413d-8a50-57152469278c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:15.666729 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 351/709] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Aug 28 00:42:15 2022] GET /identity/v3/roles/admin => generated 83 bytes in 282 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 28 00:42:15.672542 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0bc6caf-d1a7-43db-855f-dfb8b6ea8990 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:15.694677 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-6ff891b7-3813-413d-8a50-57152469278c None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:42:15.696881 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-6ff891b7-3813-413d-8a50-57152469278c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:42:15.823776 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 359/710] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:42:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Aug 28 00:42:15.842797 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0bc6caf-d1a7-43db-855f-dfb8b6ea8990 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:15.882242 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36f85c50-1bc7-4e02-b02c-33e7a6dd402b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:15.882869 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36f85c50-1bc7-4e02-b02c-33e7a6dd402b None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:15.883149 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36f85c50-1bc7-4e02-b02c-33e7a6dd402b None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:15.883413 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36f85c50-1bc7-4e02-b02c-33e7a6dd402b None admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:15.884097 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36f85c50-1bc7-4e02-b02c-33e7a6dd402b None admin] RBAC: Adding query filter params (name=admin) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:15.884398 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36f85c50-1bc7-4e02-b02c-33e7a6dd402b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:15.885846 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36f85c50-1bc7-4e02-b02c-33e7a6dd402b None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:15.896386 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 352/711] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Aug 28 00:42:15 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 28 00:42:15.901810 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ff891b7-3813-413d-8a50-57152469278c None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:16.040411 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:16.071900 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ff891b7-3813-413d-8a50-57152469278c None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:16.126609 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7c2f64a-c54e-421e-b153-9b85a78c933f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:16.127270 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7c2f64a-c54e-421e-b153-9b85a78c933f None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:16.127550 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7c2f64a-c54e-421e-b153-9b85a78c933f None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:16.127818 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7c2f64a-c54e-421e-b153-9b85a78c933f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:16.130503 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36f85c50-1bc7-4e02-b02c-33e7a6dd402b None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:16.144191 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7c2f64a-c54e-421e-b153-9b85a78c933f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:16.145793 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7c2f64a-c54e-421e-b153-9b85a78c933f None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:16.169439 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-a7c2f64a-c54e-421e-b153-9b85a78c933f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:42:16.169439 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:16.169439 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:16.169439 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:16.169439 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:16.169439 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:16.169439 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:16.169439 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:16.169439 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:16.169439 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:16.169439 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:16.169439 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:16.169439 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 28 00:42:16.169439 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 28 00:42:16.169439 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 28 00:42:16.169439 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 28 00:42:16.170465 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:16.170465 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:16.170465 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:16.170465 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:16.170465 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:16.170465 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:16.170465 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:16.170465 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:16.170465 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:16.170465 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:16.170465 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:16.170465 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:16.170465 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:16.170465 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:16.170465 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:16.170465 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:16.170465 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 28 00:42:16.171458 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 28 00:42:16.171458 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:42:16.171458 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:16.171458 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 360/712] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:42:15 2022] GET /identity/v3/domains/Default => generated 87 bytes in 269 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:16.178172 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7c2f64a-c54e-421e-b153-9b85a78c933f None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:16.197389 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:16.243755 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36f85c50-1bc7-4e02-b02c-33e7a6dd402b None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:16.280448 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3c4dd7e-d15a-414e-9ae6-35e0c831266e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:16.282039 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3c4dd7e-d15a-414e-9ae6-35e0c831266e None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:16.282333 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3c4dd7e-d15a-414e-9ae6-35e0c831266e None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:16.282607 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3c4dd7e-d15a-414e-9ae6-35e0c831266e None admin] PATH_INFO: `/v3/roles/service` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:16.306896 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3c4dd7e-d15a-414e-9ae6-35e0c831266e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 28 00:42:16.323114 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-d3c4dd7e-d15a-414e-9ae6-35e0c831266e None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 28 00:42:16.323114 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:16.323114 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:16.323114 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:16.323114 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:16.323114 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:16.323114 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:16.323114 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:16.323114 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:16.323114 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:16.323114 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:16.323114 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:16.323114 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 28 00:42:16.323114 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 28 00:42:16.323114 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 28 00:42:16.323114 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 28 00:42:16.324170 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 28 00:42:16.324170 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 28 00:42:16.324170 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 28 00:42:16.324170 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 28 00:42:16.324170 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:16.324170 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:16.324170 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:16.324170 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:16.324170 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:16.324170 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:16.324170 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:16.324170 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:16.324170 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:16.324170 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:16.324170 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:16.324170 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:16.325024 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:16.325024 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:16.325024 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:16.325024 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:16.325024 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 28 00:42:16.325024 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 28 00:42:16.325024 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 28 00:42:16.325024 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 28 00:42:16.325024 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 28 00:42:16.325024 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 28 00:42:16.325024 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 28 00:42:16.325024 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:42:16.325024 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 353/713] 10.222.0.42 () {56 vars in 1102 bytes} [Sun Aug 28 00:42:16 2022] GET /identity/v3/roles/service => generated 85 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:16.344199 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3c4dd7e-d15a-414e-9ae6-35e0c831266e None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:16.378961 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7c2f64a-c54e-421e-b153-9b85a78c933f None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:16.421210 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baa4c6ac-6639-44fc-a12c-8951aaaa772a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:16.422143 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa4c6ac-6639-44fc-a12c-8951aaaa772a None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:16.422622 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa4c6ac-6639-44fc-a12c-8951aaaa772a None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:16.423010 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa4c6ac-6639-44fc-a12c-8951aaaa772a None admin] PATH_INFO: `/v3/domains` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:16.423670 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baa4c6ac-6639-44fc-a12c-8951aaaa772a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:16.424133 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baa4c6ac-6639-44fc-a12c-8951aaaa772a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:16.426109 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baa4c6ac-6639-44fc-a12c-8951aaaa772a None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:16.427919 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:16.442836 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 361/714] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Aug 28 00:42:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 265 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 28 00:42:16.444125 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e4d87bc-cde1-4fe5-81f6-d7593df3459d None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:16.444409 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e4d87bc-cde1-4fe5-81f6-d7593df3459d None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:16.444685 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e4d87bc-cde1-4fe5-81f6-d7593df3459d None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:16.446642 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 362/715] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:42:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:42:16.447821 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e4d87bc-cde1-4fe5-81f6-d7593df3459d None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:16.479514 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3c4dd7e-d15a-414e-9ae6-35e0c831266e None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:16.498485 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:16.517544 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c513310a-0b59-466b-8412-6c54fdfc7d76 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:16.518299 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c513310a-0b59-466b-8412-6c54fdfc7d76 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:16.518678 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c513310a-0b59-466b-8412-6c54fdfc7d76 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:16.519045 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c513310a-0b59-466b-8412-6c54fdfc7d76 None admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:16.519802 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c513310a-0b59-466b-8412-6c54fdfc7d76 None admin] RBAC: Adding query filter params (name=service) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:16.520199 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c513310a-0b59-466b-8412-6c54fdfc7d76 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:16.521725 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c513310a-0b59-466b-8412-6c54fdfc7d76 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:16.535224 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 354/716] 10.222.0.42 () {56 vars in 1103 bytes} [Sun Aug 28 00:42:16 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 28 00:42:16.536649 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8287bd8-af4f-48e5-bc0e-549a2fc86fda None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:16.537077 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8287bd8-af4f-48e5-bc0e-549a2fc86fda None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:16.537536 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8287bd8-af4f-48e5-bc0e-549a2fc86fda None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:16.555465 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-e8287bd8-af4f-48e5-bc0e-549a2fc86fda None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:42:16.556789 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-e8287bd8-af4f-48e5-bc0e-549a2fc86fda None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:42:16.593150 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e4d87bc-cde1-4fe5-81f6-d7593df3459d None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:16.638264 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 355/717] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:42:16 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 28 00:42:16.639324 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8287bd8-af4f-48e5-bc0e-549a2fc86fda None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:16.645870 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cab72f68-b345-4a97-b33d-f54ec011c31e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:16.647327 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cab72f68-b345-4a97-b33d-f54ec011c31e None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:16.647610 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cab72f68-b345-4a97-b33d-f54ec011c31e None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:16.647882 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cab72f68-b345-4a97-b33d-f54ec011c31e None admin] PATH_INFO: `/v3/users/nova` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:16.663997 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cab72f68-b345-4a97-b33d-f54ec011c31e None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:16.665513 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cab72f68-b345-4a97-b33d-f54ec011c31e None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:16.687012 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-cab72f68-b345-4a97-b33d-f54ec011c31e None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 28 00:42:16.687012 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:16.687012 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:16.687012 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:16.687012 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:16.687012 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:16.687012 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:16.687012 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:16.687012 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:16.687012 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:16.687012 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:16.687012 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:16.687012 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 28 00:42:16.687012 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 28 00:42:16.687012 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 28 00:42:16.687012 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 28 00:42:16.689086 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:16.689086 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:16.689086 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 28 00:42:16.689086 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 28 00:42:16.689086 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 28 00:42:16.689086 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 28 00:42:16.689086 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:16.689086 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:16.689086 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:16.689086 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:16.689086 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:16.689086 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:16.689086 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:16.689086 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:16.689086 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:16.689086 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:16.689086 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:16.689086 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:16.691163 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:16.691163 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:16.691163 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 28 00:42:16.691163 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 28 00:42:16.691163 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 28 00:42:16.691163 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 28 00:42:16.691163 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 28 00:42:16.691163 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 28 00:42:16.691163 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 28 00:42:16.691163 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:16.694650 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 363/718] 10.222.0.42 () {56 vars in 1093 bytes} [Sun Aug 28 00:42:16 2022] GET /identity/v3/users/nova => generated 82 bytes in 241 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:16.694650 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cab72f68-b345-4a97-b33d-f54ec011c31e None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:16.767724 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:16.819363 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cab72f68-b345-4a97-b33d-f54ec011c31e None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:16.858763 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb52803e-e8ac-40e8-896f-81ef30f462cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:16.859383 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb52803e-e8ac-40e8-896f-81ef30f462cd None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:16.859664 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb52803e-e8ac-40e8-896f-81ef30f462cd None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:16.859938 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb52803e-e8ac-40e8-896f-81ef30f462cd None admin] PATH_INFO: `/v3/users` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:16.860674 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb52803e-e8ac-40e8-896f-81ef30f462cd None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:16.861001 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb52803e-e8ac-40e8-896f-81ef30f462cd None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:16.862632 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb52803e-e8ac-40e8-896f-81ef30f462cd None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:16.863104 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8287bd8-af4f-48e5-bc0e-549a2fc86fda None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:16.882582 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 364/719] 10.222.0.42 () {56 vars in 1133 bytes} [Sun Aug 28 00:42:16 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:16.888246 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:16.890410 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb52803e-e8ac-40e8-896f-81ef30f462cd None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:16.952146 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c7b8ac6-408c-41e5-ba55-2aa16c8bd242 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:16.952789 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c7b8ac6-408c-41e5-ba55-2aa16c8bd242 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:16.953091 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c7b8ac6-408c-41e5-ba55-2aa16c8bd242 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:16.953359 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c7b8ac6-408c-41e5-ba55-2aa16c8bd242 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:16.968106 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c7b8ac6-408c-41e5-ba55-2aa16c8bd242 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:16.970374 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c7b8ac6-408c-41e5-ba55-2aa16c8bd242 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:16.988755 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-3c7b8ac6-408c-41e5-ba55-2aa16c8bd242 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:42:16.988755 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:16.988755 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:16.988755 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:16.988755 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:16.988755 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:16.988755 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:16.988755 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:16.988755 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:16.988755 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:16.988755 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:16.988755 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:16.988755 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 28 00:42:16.988755 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 28 00:42:16.988755 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 28 00:42:16.988755 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 28 00:42:16.989981 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:16.989981 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:16.989981 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:16.989981 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:16.989981 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:16.989981 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:16.989981 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:16.989981 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:16.989981 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:16.989981 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:16.989981 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:16.989981 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:16.989981 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:16.989981 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:16.989981 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:16.989981 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:16.989981 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 28 00:42:16.991084 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 28 00:42:16.991084 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:42:16.991084 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:42:16.994833 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 356/720] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:42:16 2022] GET /identity/v3/domains/Default => generated 87 bytes in 356 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:16.995672 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c7b8ac6-408c-41e5-ba55-2aa16c8bd242 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:17.028624 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb52803e-e8ac-40e8-896f-81ef30f462cd None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:17.043379 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:17.065161 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70a59a95-292c-4ce1-8478-b3d0142745cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:17.066039 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a59a95-292c-4ce1-8478-b3d0142745cb None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:17.066477 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a59a95-292c-4ce1-8478-b3d0142745cb None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:17.066893 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a59a95-292c-4ce1-8478-b3d0142745cb None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:17.100828 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c7b8ac6-408c-41e5-ba55-2aa16c8bd242 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:17.138115 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70a59a95-292c-4ce1-8478-b3d0142745cb None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:17.138315 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65968457-eac6-427c-b4b8-522de7fea783 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:17.138930 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65968457-eac6-427c-b4b8-522de7fea783 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:17.139211 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65968457-eac6-427c-b4b8-522de7fea783 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:17.139514 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65968457-eac6-427c-b4b8-522de7fea783 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:17.140685 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70a59a95-292c-4ce1-8478-b3d0142745cb None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:17.150732 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65968457-eac6-427c-b4b8-522de7fea783 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 28 00:42:17.156508 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-65968457-eac6-427c-b4b8-522de7fea783 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 28 00:42:17.156508 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:17.156508 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:17.156508 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:17.156508 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:17.156508 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:17.156508 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:17.156508 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:17.156508 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:17.156508 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:17.156508 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:17.156508 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:17.156508 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 28 00:42:17.156508 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 28 00:42:17.156508 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 28 00:42:17.156508 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 28 00:42:17.157687 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 28 00:42:17.157687 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 28 00:42:17.157687 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 28 00:42:17.157687 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 28 00:42:17.157687 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:17.157687 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:17.157687 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:17.157687 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:17.157687 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:17.157687 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:17.157687 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:17.157687 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:17.157687 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:17.157687 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:17.157687 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:17.157687 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:17.158701 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:17.158701 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:17.158701 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:17.158701 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:17.158701 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 28 00:42:17.158701 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 28 00:42:17.158701 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 28 00:42:17.158701 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 28 00:42:17.158701 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 28 00:42:17.158701 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 28 00:42:17.158701 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 28 00:42:17.158701 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:42:17.162608 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 357/721] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Aug 28 00:42:16 2022] GET /identity/v3/roles/admin => generated 83 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:17.163602 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65968457-eac6-427c-b4b8-522de7fea783 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:17.209398 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:17.247365 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 365/722] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Aug 28 00:42:16 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 358 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 28 00:42:17.248565 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70a59a95-292c-4ce1-8478-b3d0142745cb None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:17.262863 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:17.272522 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65968457-eac6-427c-b4b8-522de7fea783 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:17.308454 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8080c059-93f0-41a7-b4f5-5061cbad386c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:17.309174 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8080c059-93f0-41a7-b4f5-5061cbad386c None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:17.309744 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8080c059-93f0-41a7-b4f5-5061cbad386c None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:17.309854 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8080c059-93f0-41a7-b4f5-5061cbad386c None admin] PATH_INFO: `/v3/domains` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:17.310432 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8080c059-93f0-41a7-b4f5-5061cbad386c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:17.310734 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8080c059-93f0-41a7-b4f5-5061cbad386c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:17.312130 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8080c059-93f0-41a7-b4f5-5061cbad386c None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:17.329503 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 358/723] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Aug 28 00:42:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:17.330835 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8080c059-93f0-41a7-b4f5-5061cbad386c None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:17.354149 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70a59a95-292c-4ce1-8478-b3d0142745cb None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:17.389301 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09eabc8b-3f5a-4c50-9505-36520c03d781 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:17.390283 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09eabc8b-3f5a-4c50-9505-36520c03d781 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:17.390797 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09eabc8b-3f5a-4c50-9505-36520c03d781 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:17.391247 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09eabc8b-3f5a-4c50-9505-36520c03d781 None admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:17.392081 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09eabc8b-3f5a-4c50-9505-36520c03d781 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:17.392575 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09eabc8b-3f5a-4c50-9505-36520c03d781 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:17.394284 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09eabc8b-3f5a-4c50-9505-36520c03d781 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:17.403338 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 366/724] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Aug 28 00:42:17 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:17.404438 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09eabc8b-3f5a-4c50-9505-36520c03d781 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:17.422932 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:17.444305 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8080c059-93f0-41a7-b4f5-5061cbad386c None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:17.485207 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:17.487072 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ef6b35e-8371-4d58-a13c-ba5a424a1bc5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:17.488704 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ef6b35e-8371-4d58-a13c-ba5a424a1bc5 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:17.489075 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ef6b35e-8371-4d58-a13c-ba5a424a1bc5 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:17.489385 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ef6b35e-8371-4d58-a13c-ba5a424a1bc5 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:17.500234 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ef6b35e-8371-4d58-a13c-ba5a424a1bc5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:17.501705 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ef6b35e-8371-4d58-a13c-ba5a424a1bc5 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:17.512844 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09eabc8b-3f5a-4c50-9505-36520c03d781 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:17.516439 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-6ef6b35e-8371-4d58-a13c-ba5a424a1bc5 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:42:17.516439 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:17.516439 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:17.516439 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:17.516439 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:17.516439 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:17.516439 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:17.516439 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:17.516439 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:17.516439 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:17.516439 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:17.516439 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:17.516439 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 28 00:42:17.516439 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 28 00:42:17.516439 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 28 00:42:17.516439 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 28 00:42:17.521846 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:17.521846 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:17.521846 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:17.521846 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:17.521846 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:17.521846 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:17.521846 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:17.521846 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:17.521846 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:17.521846 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:17.521846 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:17.521846 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:17.521846 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:17.521846 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:17.521846 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:17.521846 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:17.521846 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 28 00:42:17.522774 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 28 00:42:17.522774 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:42:17.522774 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:42:17.522774 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 359/725] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:42:17 2022] GET /identity/v3/domains/Default => generated 87 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:17.522774 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ef6b35e-8371-4d58-a13c-ba5a424a1bc5 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:17.548857 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b79c242-72a1-463d-9e9d-9a69236faeef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:17.549525 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b79c242-72a1-463d-9e9d-9a69236faeef None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:17.549804 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b79c242-72a1-463d-9e9d-9a69236faeef None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:17.550079 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b79c242-72a1-463d-9e9d-9a69236faeef None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:17.559577 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b79c242-72a1-463d-9e9d-9a69236faeef None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:17.562020 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b79c242-72a1-463d-9e9d-9a69236faeef None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:17.577941 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-9b79c242-72a1-463d-9e9d-9a69236faeef None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 28 00:42:17.577941 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:17.577941 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:17.577941 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:17.577941 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:17.577941 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:17.577941 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:17.577941 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:17.577941 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:17.577941 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:17.577941 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:17.577941 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:17.577941 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 28 00:42:17.577941 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 28 00:42:17.577941 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 28 00:42:17.577941 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 28 00:42:17.578929 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:17.578929 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:17.578929 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 28 00:42:17.578929 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 28 00:42:17.578929 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 28 00:42:17.578929 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 28 00:42:17.578929 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:17.578929 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:17.578929 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:17.578929 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:17.578929 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:17.578929 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:17.578929 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:17.578929 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:17.578929 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:17.578929 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:17.578929 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:17.578929 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:17.585019 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:17.585019 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:17.585019 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 28 00:42:17.585019 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 28 00:42:17.585019 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 28 00:42:17.585019 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 28 00:42:17.585019 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 28 00:42:17.585019 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 28 00:42:17.585019 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 28 00:42:17.585019 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:17.585019 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 367/726] 10.222.0.42 () {56 vars in 1117 bytes} [Sun Aug 28 00:42:17 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:17.585019 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee19265d-91db-4dbb-aef3-73d9ce1cd2b8 None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:17.585019 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee19265d-91db-4dbb-aef3-73d9ce1cd2b8 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:17.585773 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee19265d-91db-4dbb-aef3-73d9ce1cd2b8 None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:17.585773 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 368/727] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:42:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:42:17.585773 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee19265d-91db-4dbb-aef3-73d9ce1cd2b8 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:17.641654 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ef6b35e-8371-4d58-a13c-ba5a424a1bc5 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:17.676336 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae172efa-dfe9-43c0-b92f-f5a371a17086 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:17.677005 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae172efa-dfe9-43c0-b92f-f5a371a17086 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:17.677280 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae172efa-dfe9-43c0-b92f-f5a371a17086 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:17.677985 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae172efa-dfe9-43c0-b92f-f5a371a17086 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:17.680137 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:17.689976 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae172efa-dfe9-43c0-b92f-f5a371a17086 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:17.691476 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae172efa-dfe9-43c0-b92f-f5a371a17086 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:17.691753 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee19265d-91db-4dbb-aef3-73d9ce1cd2b8 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:17.707581 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-ae172efa-dfe9-43c0-b92f-f5a371a17086 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:42:17.707581 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:17.707581 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:17.707581 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:17.707581 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:17.707581 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:17.707581 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:17.707581 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:17.707581 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:17.707581 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:17.707581 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:17.707581 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:17.707581 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 28 00:42:17.707581 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 28 00:42:17.707581 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 28 00:42:17.707581 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 28 00:42:17.708591 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:17.708591 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:17.708591 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:17.708591 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:17.708591 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:17.708591 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:17.708591 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:17.708591 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:17.708591 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:17.708591 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:17.708591 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:17.708591 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:17.708591 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:17.708591 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:17.708591 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:17.708591 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:17.708591 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 28 00:42:17.709517 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 28 00:42:17.709517 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:42:17.709517 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:42:17.709517 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 360/728] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:42:17 2022] GET /identity/v3/domains/Default => generated 87 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:17.710169 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d4f3afb-7a17-4d6e-afda-466b1d819ab8 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:17.710265 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d4f3afb-7a17-4d6e-afda-466b1d819ab8 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:17.710530 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d4f3afb-7a17-4d6e-afda-466b1d819ab8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:17.729541 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-6d4f3afb-7a17-4d6e-afda-466b1d819ab8 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:42:17.730736 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-6d4f3afb-7a17-4d6e-afda-466b1d819ab8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:42:17.731579 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c0468db-68fb-4e31-b1a6-2a9d170e7704 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:17.732197 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c0468db-68fb-4e31-b1a6-2a9d170e7704 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:17.732465 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c0468db-68fb-4e31-b1a6-2a9d170e7704 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:17.732732 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c0468db-68fb-4e31-b1a6-2a9d170e7704 None admin] PATH_INFO: `/v3/domains` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:17.733550 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c0468db-68fb-4e31-b1a6-2a9d170e7704 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:17.733839 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c0468db-68fb-4e31-b1a6-2a9d170e7704 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:17.735207 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c0468db-68fb-4e31-b1a6-2a9d170e7704 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:17.736692 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:17.748991 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 369/729] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Aug 28 00:42:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:17.749486 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c0468db-68fb-4e31-b1a6-2a9d170e7704 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:17.803643 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 361/730] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:42:17 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 28 00:42:17.804567 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d4f3afb-7a17-4d6e-afda-466b1d819ab8 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:17.861368 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c0468db-68fb-4e31-b1a6-2a9d170e7704 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:17.896039 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14df86a8-bfc4-4005-b0da-c72e07d4c8f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:17.896558 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14df86a8-bfc4-4005-b0da-c72e07d4c8f3 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:17.896835 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14df86a8-bfc4-4005-b0da-c72e07d4c8f3 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:17.897130 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14df86a8-bfc4-4005-b0da-c72e07d4c8f3 None admin] PATH_INFO: `/v3/users` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:17.897850 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14df86a8-bfc4-4005-b0da-c72e07d4c8f3 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:17.898148 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14df86a8-bfc4-4005-b0da-c72e07d4c8f3 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:17.899586 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14df86a8-bfc4-4005-b0da-c72e07d4c8f3 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:17.909377 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d4f3afb-7a17-4d6e-afda-466b1d819ab8 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:17.916872 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 370/731] 10.222.0.42 () {56 vars in 1149 bytes} [Sun Aug 28 00:42:17 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:17.917714 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14df86a8-bfc4-4005-b0da-c72e07d4c8f3 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:17.931137 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:17.952618 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47fd6bd5-0f3e-4710-b8ad-789e8acb7146 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:17.953834 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47fd6bd5-0f3e-4710-b8ad-789e8acb7146 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:17.953834 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47fd6bd5-0f3e-4710-b8ad-789e8acb7146 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:17.953834 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47fd6bd5-0f3e-4710-b8ad-789e8acb7146 None admin] PATH_INFO: `/v3/domains` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:17.954396 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47fd6bd5-0f3e-4710-b8ad-789e8acb7146 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:17.954692 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47fd6bd5-0f3e-4710-b8ad-789e8acb7146 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:17.956380 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47fd6bd5-0f3e-4710-b8ad-789e8acb7146 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:17.968439 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 362/732] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Aug 28 00:42:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:17.969955 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f390d0f-a7b9-4f85-bbf9-9ff33aef4695 None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:17.970414 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f390d0f-a7b9-4f85-bbf9-9ff33aef4695 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:17.970689 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f390d0f-a7b9-4f85-bbf9-9ff33aef4695 None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:17.976134 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 363/733] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:42:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:42:17.976134 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f390d0f-a7b9-4f85-bbf9-9ff33aef4695 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:17.980305 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:18.031113 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14df86a8-bfc4-4005-b0da-c72e07d4c8f3 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:18.068972 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b86df02b-f449-460e-ba77-9cc2dcbb7ff9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:18.069631 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b86df02b-f449-460e-ba77-9cc2dcbb7ff9 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:18.069905 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b86df02b-f449-460e-ba77-9cc2dcbb7ff9 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:18.070180 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b86df02b-f449-460e-ba77-9cc2dcbb7ff9 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:18.074091 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f390d0f-a7b9-4f85-bbf9-9ff33aef4695 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:18.078737 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b86df02b-f449-460e-ba77-9cc2dcbb7ff9 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:18.080909 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b86df02b-f449-460e-ba77-9cc2dcbb7ff9 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:18.093015 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-b86df02b-f449-460e-ba77-9cc2dcbb7ff9 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 28 00:42:18.093015 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:18.093015 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:18.093015 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:18.093015 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:18.093015 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:18.093015 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:18.093015 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:18.093015 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:18.093015 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:18.093015 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:18.093015 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:18.093015 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 28 00:42:18.093015 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 28 00:42:18.093015 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 28 00:42:18.094072 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 28 00:42:18.094072 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:18.094072 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:18.094072 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:18.094072 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:18.094072 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:18.094072 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:18.094072 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:18.094072 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:18.094072 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:18.094072 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:18.094072 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:18.094072 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:18.094072 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:18.094072 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:18.094072 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:18.094072 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:18.095126 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 28 00:42:18.095126 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 28 00:42:18.095126 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 28 00:42:18.095126 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 28 00:42:18.095126 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 28 00:42:18.095126 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 28 00:42:18.095126 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 28 00:42:18.095126 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:18.095126 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 371/734] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Aug 28 00:42:17 2022] GET /identity/v3/projects/service => generated 88 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:18.095126 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b86df02b-f449-460e-ba77-9cc2dcbb7ff9 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:18.114384 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7c74699-a1c9-43bb-b0ee-9695ac228b27 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:18.115113 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7c74699-a1c9-43bb-b0ee-9695ac228b27 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:18.115354 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7c74699-a1c9-43bb-b0ee-9695ac228b27 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:18.115621 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7c74699-a1c9-43bb-b0ee-9695ac228b27 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:18.159793 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:18.182297 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7c74699-a1c9-43bb-b0ee-9695ac228b27 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:18.183785 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7c74699-a1c9-43bb-b0ee-9695ac228b27 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:18.191901 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b86df02b-f449-460e-ba77-9cc2dcbb7ff9 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:18.204054 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:18.225516 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f20fb694-ff61-4dda-a573-28fe517d2111 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:18.226128 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f20fb694-ff61-4dda-a573-28fe517d2111 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:18.226405 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f20fb694-ff61-4dda-a573-28fe517d2111 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:18.226695 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f20fb694-ff61-4dda-a573-28fe517d2111 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:18.233971 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f20fb694-ff61-4dda-a573-28fe517d2111 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:18.235398 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f20fb694-ff61-4dda-a573-28fe517d2111 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:18.246496 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-f20fb694-ff61-4dda-a573-28fe517d2111 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Aug 28 00:42:18.246496 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:18.246496 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:18.246496 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:18.246496 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:18.246496 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:18.246496 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:18.246496 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:18.246496 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:18.246496 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:18.246496 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:18.246496 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:18.246496 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 28 00:42:18.246496 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 28 00:42:18.246496 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 28 00:42:18.246496 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 28 00:42:18.247424 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:18.247424 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:18.247424 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 28 00:42:18.247424 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 28 00:42:18.247424 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 28 00:42:18.247424 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 28 00:42:18.247424 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:18.247424 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:18.247424 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:18.247424 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:18.247424 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:18.247424 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:18.247424 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:18.247424 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:18.247424 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:18.247424 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:18.247424 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:18.247424 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:18.248347 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:18.248347 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:18.248347 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 28 00:42:18.248347 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 28 00:42:18.248347 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 28 00:42:18.248347 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 28 00:42:18.248347 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 28 00:42:18.248347 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 28 00:42:18.248347 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Aug 28 00:42:18.248347 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:18.248347 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 372/735] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Aug 28 00:42:18 2022] GET /identity/v3/users/swift => generated 83 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:18.248938 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7165d36c-abd4-4445-92fd-65539da24656 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:18.248938 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7165d36c-abd4-4445-92fd-65539da24656 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:18.249318 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7165d36c-abd4-4445-92fd-65539da24656 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:18.268706 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-7165d36c-abd4-4445-92fd-65539da24656 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:42:18.269902 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-7165d36c-abd4-4445-92fd-65539da24656 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:42:18.285649 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 364/736] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Aug 28 00:42:17 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 313 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Aug 28 00:42:18.286492 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7c74699-a1c9-43bb-b0ee-9695ac228b27 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:18.341512 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 373/737] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:42:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Aug 28 00:42:18.342770 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7165d36c-abd4-4445-92fd-65539da24656 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:18.364186 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:18.388707 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7c74699-a1c9-43bb-b0ee-9695ac228b27 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:18.421665 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb84ecab-56a8-4a92-8ed8-bcf73f9c24a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:18.422196 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb84ecab-56a8-4a92-8ed8-bcf73f9c24a9 None admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:18.422471 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb84ecab-56a8-4a92-8ed8-bcf73f9c24a9 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:18.422739 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb84ecab-56a8-4a92-8ed8-bcf73f9c24a9 None admin] PATH_INFO: `/v3/services` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:18.423383 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb84ecab-56a8-4a92-8ed8-bcf73f9c24a9 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:18.424759 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb84ecab-56a8-4a92-8ed8-bcf73f9c24a9 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:18.440418 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 365/738] 10.222.0.42 () {60 vars in 1141 bytes} [Sun Aug 28 00:42:18 2022] POST /identity/v3/services => generated 240 bytes in 154 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:42:18.441556 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb84ecab-56a8-4a92-8ed8-bcf73f9c24a9 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:18.457210 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7165d36c-abd4-4445-92fd-65539da24656 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:18.459791 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:18.513049 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee76ebde-cda2-4e3a-a72c-b112f9007500 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:18.514029 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee76ebde-cda2-4e3a-a72c-b112f9007500 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:18.514565 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee76ebde-cda2-4e3a-a72c-b112f9007500 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:18.515022 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee76ebde-cda2-4e3a-a72c-b112f9007500 None admin] PATH_INFO: `/v3/projects` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:18.516102 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee76ebde-cda2-4e3a-a72c-b112f9007500 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:18.516678 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee76ebde-cda2-4e3a-a72c-b112f9007500 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:18.518437 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee76ebde-cda2-4e3a-a72c-b112f9007500 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:18.534139 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 374/739] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Aug 28 00:42:18 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:18.538451 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee76ebde-cda2-4e3a-a72c-b112f9007500 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:18.583888 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb84ecab-56a8-4a92-8ed8-bcf73f9c24a9 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:18.630791 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0d89ca7-27ea-4b65-a5de-dfad881d915c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:18.631590 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d89ca7-27ea-4b65-a5de-dfad881d915c None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:18.631708 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d89ca7-27ea-4b65-a5de-dfad881d915c None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:18.631937 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d89ca7-27ea-4b65-a5de-dfad881d915c None admin] PATH_INFO: `/v3/users` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:18.632664 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0d89ca7-27ea-4b65-a5de-dfad881d915c None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:18.633574 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0d89ca7-27ea-4b65-a5de-dfad881d915c None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:18.634432 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0d89ca7-27ea-4b65-a5de-dfad881d915c None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:18.637097 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:18.655890 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee76ebde-cda2-4e3a-a72c-b112f9007500 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:18.657535 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 366/740] 10.222.0.42 () {56 vars in 1135 bytes} [Sun Aug 28 00:42:18 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:18.658387 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0d89ca7-27ea-4b65-a5de-dfad881d915c None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:18.695813 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42d61586-cad4-4748-a4f4-6757935c95e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:18.696791 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42d61586-cad4-4748-a4f4-6757935c95e9 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:18.697316 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42d61586-cad4-4748-a4f4-6757935c95e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:18.697784 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42d61586-cad4-4748-a4f4-6757935c95e9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:18.709034 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42d61586-cad4-4748-a4f4-6757935c95e9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:18.710628 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42d61586-cad4-4748-a4f4-6757935c95e9 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:18.725564 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-42d61586-cad4-4748-a4f4-6757935c95e9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:42:18.725564 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:18.725564 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:18.725564 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:18.725564 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:18.725564 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:18.725564 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:18.725564 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:18.725564 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:18.725564 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:18.725564 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:18.725564 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:18.725564 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 28 00:42:18.725564 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 28 00:42:18.725564 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 28 00:42:18.725564 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 28 00:42:18.726523 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:18.726523 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:18.726523 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:18.726523 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:18.726523 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:18.726523 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:18.726523 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:18.726523 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:18.726523 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:18.726523 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:18.726523 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:18.726523 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:18.726523 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:18.726523 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:18.726523 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:18.726523 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:18.726523 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 28 00:42:18.727436 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 28 00:42:18.727436 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:42:18.727436 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:18.727436 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 375/741] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:42:18 2022] GET /identity/v3/domains/Default => generated 87 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:18.727436 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42d61586-cad4-4748-a4f4-6757935c95e9 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:18.741750 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:18.770813 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0d89ca7-27ea-4b65-a5de-dfad881d915c None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:18.806576 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1621a45-39b0-4203-bde6-ef2c6911a752 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:18.807190 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1621a45-39b0-4203-bde6-ef2c6911a752 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:18.807468 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1621a45-39b0-4203-bde6-ef2c6911a752 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:18.807733 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1621a45-39b0-4203-bde6-ef2c6911a752 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:18.808460 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1621a45-39b0-4203-bde6-ef2c6911a752 None admin] RBAC: Adding query filter params (role.id=f08d6043261747e3b4763c744eef7dfb, scope.project.id=f53e6fdb369b480fbebb4086a505df67, user.id=168b07df50c24593ba73dbde5b037aed) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:18.808766 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1621a45-39b0-4203-bde6-ef2c6911a752 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:18.810251 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1621a45-39b0-4203-bde6-ef2c6911a752 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:18.821473 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 367/742] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Aug 28 00:42:18 2022] GET /identity/v3/role_assignments?user.id=168b07df50c24593ba73dbde5b037aed&scope.project.id=f53e6fdb369b480fbebb4086a505df67&role.id=f08d6043261747e3b4763c744eef7dfb => generated 259 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:18.822482 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1621a45-39b0-4203-bde6-ef2c6911a752 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:18.858520 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42d61586-cad4-4748-a4f4-6757935c95e9 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:18.908418 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fab02772-f58b-4659-bad9-1af5afcd497c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:18.909161 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fab02772-f58b-4659-bad9-1af5afcd497c None admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:18.909451 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fab02772-f58b-4659-bad9-1af5afcd497c None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:18.909722 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fab02772-f58b-4659-bad9-1af5afcd497c None admin] PATH_INFO: `/v3/services` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:18.910459 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fab02772-f58b-4659-bad9-1af5afcd497c None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:18.911915 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fab02772-f58b-4659-bad9-1af5afcd497c None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:18.913894 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:18.933627 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 376/743] 10.222.0.42 () {60 vars in 1141 bytes} [Sun Aug 28 00:42:18 2022] POST /identity/v3/services => generated 251 bytes in 206 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:42:18.936093 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fab02772-f58b-4659-bad9-1af5afcd497c None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:18.973930 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1621a45-39b0-4203-bde6-ef2c6911a752 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:19.021949 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90ecdb29-f1a5-46a3-a1c6-853b00e976c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:19.022848 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90ecdb29-f1a5-46a3-a1c6-853b00e976c0 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:19.023285 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90ecdb29-f1a5-46a3-a1c6-853b00e976c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:19.023702 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90ecdb29-f1a5-46a3-a1c6-853b00e976c0 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:19.035571 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:19.121315 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90ecdb29-f1a5-46a3-a1c6-853b00e976c0 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:19.122551 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90ecdb29-f1a5-46a3-a1c6-853b00e976c0 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:19.197666 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:19.221220 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fab02772-f58b-4659-bad9-1af5afcd497c None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:19.262729 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0358f42-6516-4722-839f-462112bbefc9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:19.263284 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0358f42-6516-4722-839f-462112bbefc9 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:19.263595 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0358f42-6516-4722-839f-462112bbefc9 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:19.263857 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0358f42-6516-4722-839f-462112bbefc9 None admin] PATH_INFO: `/v3/domains` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:19.264634 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0358f42-6516-4722-839f-462112bbefc9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:19.264872 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0358f42-6516-4722-839f-462112bbefc9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:19.266409 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0358f42-6516-4722-839f-462112bbefc9 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:19.279217 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 377/744] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Aug 28 00:42:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 345 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:19.288859 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 368/745] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Aug 28 00:42:18 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 467 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 28 00:42:19.290182 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0358f42-6516-4722-839f-462112bbefc9 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:19.294441 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90ecdb29-f1a5-46a3-a1c6-853b00e976c0 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:19.426093 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0358f42-6516-4722-839f-462112bbefc9 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:19.488720 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:19.500158 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6298ce8a-76b9-4f88-ac2a-0cd95efd6ec3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:19.500785 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6298ce8a-76b9-4f88-ac2a-0cd95efd6ec3 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:19.501081 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6298ce8a-76b9-4f88-ac2a-0cd95efd6ec3 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:19.501347 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6298ce8a-76b9-4f88-ac2a-0cd95efd6ec3 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:19.503680 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90ecdb29-f1a5-46a3-a1c6-853b00e976c0 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:19.518429 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6298ce8a-76b9-4f88-ac2a-0cd95efd6ec3 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:19.521737 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6298ce8a-76b9-4f88-ac2a-0cd95efd6ec3 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:19.541303 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-6298ce8a-76b9-4f88-ac2a-0cd95efd6ec3 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 28 00:42:19.541303 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:19.541303 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:19.541303 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:19.541303 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:19.541303 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:19.541303 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:19.541303 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:19.541303 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:19.541303 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:19.541303 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:19.541303 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:19.541303 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 28 00:42:19.541303 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 28 00:42:19.541303 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 28 00:42:19.542260 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 28 00:42:19.542260 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:19.542260 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:19.542260 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:19.542260 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:19.542260 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:19.542260 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:19.542260 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:19.542260 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:19.542260 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:19.542260 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:19.542260 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:19.542260 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:19.542260 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:19.542260 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:19.542260 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:19.542260 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:19.543152 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 28 00:42:19.543152 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 28 00:42:19.543152 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 28 00:42:19.543152 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 28 00:42:19.543152 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 28 00:42:19.543152 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 28 00:42:19.543152 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 28 00:42:19.543152 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:19.543152 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 378/746] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Aug 28 00:42:19 2022] GET /identity/v3/projects/service => generated 88 bytes in 253 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:19.548558 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6298ce8a-76b9-4f88-ac2a-0cd95efd6ec3 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:19.613899 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73d0862a-d84d-4a6f-8830-58b09be721d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:19.614539 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73d0862a-d84d-4a6f-8830-58b09be721d1 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:19.614812 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73d0862a-d84d-4a6f-8830-58b09be721d1 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:19.615079 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73d0862a-d84d-4a6f-8830-58b09be721d1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:19.627962 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73d0862a-d84d-4a6f-8830-58b09be721d1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:19.629510 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73d0862a-d84d-4a6f-8830-58b09be721d1 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:19.644570 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-73d0862a-d84d-4a6f-8830-58b09be721d1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:42:19.644570 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:19.644570 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:19.644570 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:19.644570 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:19.644570 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:19.644570 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:19.644570 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:19.644570 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:19.644570 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:19.644570 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:19.644570 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:19.644570 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 28 00:42:19.644570 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 28 00:42:19.644570 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 28 00:42:19.644570 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 28 00:42:19.645570 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:19.645570 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:19.645570 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:19.645570 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:19.645570 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:19.645570 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:19.645570 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:19.645570 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:19.645570 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:19.645570 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:19.645570 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:19.645570 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:19.645570 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:19.645570 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:19.645570 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:19.645570 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:19.645570 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 28 00:42:19.646467 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 28 00:42:19.646467 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:42:19.646467 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:42:19.646467 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 369/747] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:42:19 2022] GET /identity/v3/domains/Default => generated 87 bytes in 352 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:19.651390 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73d0862a-d84d-4a6f-8830-58b09be721d1 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:19.661419 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:19.710857 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6298ce8a-76b9-4f88-ac2a-0cd95efd6ec3 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:19.745752 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29a8cfb1-89e3-4a27-b0c1-8742588518ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:19.746385 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29a8cfb1-89e3-4a27-b0c1-8742588518ee None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:19.746627 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29a8cfb1-89e3-4a27-b0c1-8742588518ee None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:19.746899 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29a8cfb1-89e3-4a27-b0c1-8742588518ee None admin] PATH_INFO: `/v3/projects` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:19.747865 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29a8cfb1-89e3-4a27-b0c1-8742588518ee None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:19.748170 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29a8cfb1-89e3-4a27-b0c1-8742588518ee None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:19.749851 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29a8cfb1-89e3-4a27-b0c1-8742588518ee None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:19.764559 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 379/748] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Aug 28 00:42:19 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:19.771458 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29a8cfb1-89e3-4a27-b0c1-8742588518ee None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:19.812425 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73d0862a-d84d-4a6f-8830-58b09be721d1 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:19.847473 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4240d43-f7fe-4b49-b811-b87d0342fb1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:19.848466 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4240d43-f7fe-4b49-b811-b87d0342fb1b None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:19.848981 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4240d43-f7fe-4b49-b811-b87d0342fb1b None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:19.849448 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4240d43-f7fe-4b49-b811-b87d0342fb1b None admin] PATH_INFO: `/v3/domains` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:19.850302 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4240d43-f7fe-4b49-b811-b87d0342fb1b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:19.850857 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4240d43-f7fe-4b49-b811-b87d0342fb1b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:19.852443 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4240d43-f7fe-4b49-b811-b87d0342fb1b None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:19.865739 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 370/749] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Aug 28 00:42:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 28 00:42:19.869890 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:19.871852 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4240d43-f7fe-4b49-b811-b87d0342fb1b None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:19.945195 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:19.962100 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29a8cfb1-89e3-4a27-b0c1-8742588518ee None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:19.985472 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4240d43-f7fe-4b49-b811-b87d0342fb1b None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:20.024903 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8408680b-8d3e-423f-bfaf-245df24fb083 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:20.025671 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8408680b-8d3e-423f-bfaf-245df24fb083 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:20.025949 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8408680b-8d3e-423f-bfaf-245df24fb083 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:20.026263 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8408680b-8d3e-423f-bfaf-245df24fb083 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:20.034283 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8408680b-8d3e-423f-bfaf-245df24fb083 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:20.035814 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8408680b-8d3e-423f-bfaf-245df24fb083 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:20.041715 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c94ce4fd-3ca5-4f0a-9347-47509112e1c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:20.042344 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c94ce4fd-3ca5-4f0a-9347-47509112e1c0 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:20.042627 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c94ce4fd-3ca5-4f0a-9347-47509112e1c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:20.042900 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c94ce4fd-3ca5-4f0a-9347-47509112e1c0 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:20.043636 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c94ce4fd-3ca5-4f0a-9347-47509112e1c0 None admin] RBAC: Adding query filter params (role.id=348070fea3dd4cc2a6e00e29fe82cc25, scope.project.id=f53e6fdb369b480fbebb4086a505df67, user.id=eede08f020e04282bb994b855e371433) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:20.044030 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c94ce4fd-3ca5-4f0a-9347-47509112e1c0 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:20.045542 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c94ce4fd-3ca5-4f0a-9347-47509112e1c0 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:20.047730 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-8408680b-8d3e-423f-bfaf-245df24fb083 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 28 00:42:20.047730 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:20.047730 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:20.047730 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:20.047730 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:20.047730 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:20.047730 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:20.047730 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:20.047730 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:20.047730 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:20.047730 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:20.047730 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:20.047730 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 28 00:42:20.047730 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 28 00:42:20.047730 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 28 00:42:20.048595 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 28 00:42:20.048595 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:20.048595 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:20.048595 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:20.048595 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:20.048595 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:20.048595 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:20.048595 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:20.048595 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:20.048595 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:20.048595 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:20.048595 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:20.048595 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:20.048595 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:20.048595 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:20.048595 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:20.048595 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:20.049532 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 28 00:42:20.049532 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 28 00:42:20.049532 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 28 00:42:20.049532 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 28 00:42:20.049532 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 28 00:42:20.049532 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 28 00:42:20.049532 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 28 00:42:20.049532 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:42:20.049532 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 371/750] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Aug 28 00:42:19 2022] GET /identity/v3/projects/service => generated 88 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 28 00:42:20.056555 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8408680b-8d3e-423f-bfaf-245df24fb083 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:20.061150 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 380/751] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Aug 28 00:42:19 2022] GET /identity/v3/role_assignments?user.id=eede08f020e04282bb994b855e371433&scope.project.id=f53e6fdb369b480fbebb4086a505df67&role.id=348070fea3dd4cc2a6e00e29fe82cc25 => generated 259 bytes in 290 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:20.179528 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8408680b-8d3e-423f-bfaf-245df24fb083 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:20.209343 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:20.218529 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-530d9177-b780-4da8-a818-d7e18bab49c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:20.219159 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-530d9177-b780-4da8-a818-d7e18bab49c8 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:20.219439 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-530d9177-b780-4da8-a818-d7e18bab49c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:20.219713 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-530d9177-b780-4da8-a818-d7e18bab49c8 None admin] PATH_INFO: `/v3/projects` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:20.220501 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-530d9177-b780-4da8-a818-d7e18bab49c8 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:20.220808 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-530d9177-b780-4da8-a818-d7e18bab49c8 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:20.222344 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-530d9177-b780-4da8-a818-d7e18bab49c8 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:20.235721 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 372/752] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Aug 28 00:42:20 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:20.241785 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c94ce4fd-3ca5-4f0a-9347-47509112e1c0 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:20.367761 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c94ce4fd-3ca5-4f0a-9347-47509112e1c0 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:20.382287 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:20.404592 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5b13050-0b86-44a9-866a-9ac0ee9840ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:20.405262 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5b13050-0b86-44a9-866a-9ac0ee9840ce None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:20.405811 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5b13050-0b86-44a9-866a-9ac0ee9840ce None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:20.406131 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5b13050-0b86-44a9-866a-9ac0ee9840ce None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:20.406857 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5b13050-0b86-44a9-866a-9ac0ee9840ce None admin] RBAC: Adding query filter params (role.id=f08d6043261747e3b4763c744eef7dfb, scope.project.id=f53e6fdb369b480fbebb4086a505df67, user.id=ac4c5584582f4f55a7085af4cbbabf9b) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:20.407160 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5b13050-0b86-44a9-866a-9ac0ee9840ce None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:20.408590 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5b13050-0b86-44a9-866a-9ac0ee9840ce None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:20.414626 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 381/753] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Aug 28 00:42:20 2022] GET /identity/v3/role_assignments?user.id=ac4c5584582f4f55a7085af4cbbabf9b&scope.project.id=f53e6fdb369b480fbebb4086a505df67&role.id=f08d6043261747e3b4763c744eef7dfb => generated 259 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:20.716214 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1647527c-43bb-4d57-b4e2-0fb2553ea076 None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:20.716735 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1647527c-43bb-4d57-b4e2-0fb2553ea076 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:20.716735 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1647527c-43bb-4d57-b4e2-0fb2553ea076 None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:20.718433 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 373/754] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:42:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:42:20.723994 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a8e672-211c-47a1-9a40-1c6410294ff1 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:20.724281 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a8e672-211c-47a1-9a40-1c6410294ff1 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:20.724553 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a8e672-211c-47a1-9a40-1c6410294ff1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:20.743048 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-f1a8e672-211c-47a1-9a40-1c6410294ff1 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:42:20.744236 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-f1a8e672-211c-47a1-9a40-1c6410294ff1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:42:20.823125 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 382/755] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:42:20 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 28 00:42:21.131512 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f67d448c-d84e-42b1-9607-101711f0f587 None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:21.131795 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f67d448c-d84e-42b1-9607-101711f0f587 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:21.132068 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f67d448c-d84e-42b1-9607-101711f0f587 None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:21.133846 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 374/756] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:42:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:42:21.139043 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2996403-97e8-4ac0-93f0-3a807512f50a None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:21.139941 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2996403-97e8-4ac0-93f0-3a807512f50a None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:21.140633 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2996403-97e8-4ac0-93f0-3a807512f50a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:21.158765 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-f2996403-97e8-4ac0-93f0-3a807512f50a None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:42:21.160017 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-f2996403-97e8-4ac0-93f0-3a807512f50a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:42:21.238124 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f67d448c-d84e-42b1-9607-101711f0f587 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:21.258580 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 383/757] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:42:21 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 28 00:42:21.382609 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f67d448c-d84e-42b1-9607-101711f0f587 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:21.416531 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b42b628d-b054-4b9d-9d1d-a33ddde3c99e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:21.417195 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b42b628d-b054-4b9d-9d1d-a33ddde3c99e None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:21.417764 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b42b628d-b054-4b9d-9d1d-a33ddde3c99e None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:21.418036 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b42b628d-b054-4b9d-9d1d-a33ddde3c99e None admin] PATH_INFO: `/v3/services/network` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:21.418622 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b42b628d-b054-4b9d-9d1d-a33ddde3c99e None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:21.419994 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b42b628d-b054-4b9d-9d1d-a33ddde3c99e None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:21.431195 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-b42b628d-b054-4b9d-9d1d-a33ddde3c99e None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Aug 28 00:42:21.431195 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:21.431195 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 28 00:42:21.431195 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 28 00:42:21.431195 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 28 00:42:21.431195 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 28 00:42:21.431195 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 28 00:42:21.431195 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 28 00:42:21.431195 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Aug 28 00:42:21.431195 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:42:21.431195 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 28 00:42:21.431195 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:42:21.431195 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:21.431195 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:21.431195 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:21.431195 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:21.431195 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:21.432184 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:21.432184 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:21.432184 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:21.432184 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:21.432184 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:21.432184 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:21.432184 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 28 00:42:21.432184 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 28 00:42:21.432184 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 28 00:42:21.432184 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 28 00:42:21.432184 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:21.432184 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:21.432184 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:21.432184 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:21.432184 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:21.432184 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:21.433041 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:21.433041 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:21.433041 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:21.433041 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:21.433041 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:21.433041 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:21.433041 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:21.433041 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:21.433041 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:21.433041 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:21.433041 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 28 00:42:21.433041 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 28 00:42:21.433041 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Aug 28 00:42:21.433041 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:42:21.433041 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 375/758] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Aug 28 00:42:21 2022] GET /identity/v3/services/network => generated 88 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:21.438282 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2996403-97e8-4ac0-93f0-3a807512f50a None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:21.469317 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f2c5cd9-9cbb-463d-b7f2-ebaca0aecd31 None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:21.469659 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f2c5cd9-9cbb-463d-b7f2-ebaca0aecd31 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:21.469874 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f2c5cd9-9cbb-463d-b7f2-ebaca0aecd31 None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:21.471509 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 376/759] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:42:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:42:21.478300 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f5ecf76-7584-4dc1-b439-a05a0fd1bb2d None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:21.478591 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f5ecf76-7584-4dc1-b439-a05a0fd1bb2d None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:21.478988 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f5ecf76-7584-4dc1-b439-a05a0fd1bb2d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:21.500347 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-7f5ecf76-7584-4dc1-b439-a05a0fd1bb2d None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:42:21.501981 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-7f5ecf76-7584-4dc1-b439-a05a0fd1bb2d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:42:21.512079 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:21.558988 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:21.582868 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 377/760] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:42:21 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 28 00:42:21.583739 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f5ecf76-7584-4dc1-b439-a05a0fd1bb2d None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:21.607828 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2996403-97e8-4ac0-93f0-3a807512f50a None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:21.656148 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d912fc10-2389-4b73-ac1d-22a4c142135e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:21.656734 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d912fc10-2389-4b73-ac1d-22a4c142135e None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:21.657041 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d912fc10-2389-4b73-ac1d-22a4c142135e None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:21.657317 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d912fc10-2389-4b73-ac1d-22a4c142135e None admin] PATH_INFO: `/v3/services` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:21.657954 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d912fc10-2389-4b73-ac1d-22a4c142135e None admin] RBAC: Adding query filter params (name=network) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:21.658257 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d912fc10-2389-4b73-ac1d-22a4c142135e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:21.659753 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d912fc10-2389-4b73-ac1d-22a4c142135e None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:21.664855 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 384/761] 10.222.0.42 () {56 vars in 1112 bytes} [Sun Aug 28 00:42:21 2022] GET /identity/v3/services?name=network => generated 124 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:21.670894 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d912fc10-2389-4b73-ac1d-22a4c142135e None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:21.760154 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f5ecf76-7584-4dc1-b439-a05a0fd1bb2d None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:21.801424 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f7265b0-5c0a-4b97-9255-b966ec56c9d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:21.802061 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f7265b0-5c0a-4b97-9255-b966ec56c9d4 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:21.802341 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f7265b0-5c0a-4b97-9255-b966ec56c9d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:21.802617 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f7265b0-5c0a-4b97-9255-b966ec56c9d4 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:21.828150 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f7265b0-5c0a-4b97-9255-b966ec56c9d4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 28 00:42:21.833800 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-9f7265b0-5c0a-4b97-9255-b966ec56c9d4 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 28 00:42:21.833800 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:21.833800 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:21.833800 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:21.833800 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:21.833800 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:21.833800 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:21.833800 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:21.833800 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:21.833800 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:21.833800 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:21.833800 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:21.833800 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 28 00:42:21.833800 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 28 00:42:21.833800 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 28 00:42:21.833800 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 28 00:42:21.834758 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 28 00:42:21.834758 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 28 00:42:21.834758 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 28 00:42:21.834758 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 28 00:42:21.834758 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:21.834758 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:21.834758 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:21.834758 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:21.834758 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:21.834758 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:21.834758 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:21.834758 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:21.834758 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:21.834758 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:21.834758 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:21.834758 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:21.835590 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:21.835590 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:21.835590 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:21.835590 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:21.835590 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 28 00:42:21.835590 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 28 00:42:21.835590 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 28 00:42:21.835590 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 28 00:42:21.835590 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 28 00:42:21.835590 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 28 00:42:21.835590 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 28 00:42:21.835590 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:42:21.835590 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 378/762] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Aug 28 00:42:21 2022] GET /identity/v3/roles/admin => generated 83 bytes in 252 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:21.841597 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f7265b0-5c0a-4b97-9255-b966ec56c9d4 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:21.894639 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:21.914435 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d912fc10-2389-4b73-ac1d-22a4c142135e None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:21.916806 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:21.951940 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87841754-1b3c-4126-a0fd-6ac0a5ba183f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:21.952927 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87841754-1b3c-4126-a0fd-6ac0a5ba183f None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:21.953463 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87841754-1b3c-4126-a0fd-6ac0a5ba183f None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:21.953929 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87841754-1b3c-4126-a0fd-6ac0a5ba183f None admin] PATH_INFO: `/v3/services` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:21.955069 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87841754-1b3c-4126-a0fd-6ac0a5ba183f None admin] RBAC: Adding query filter params (type=network) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:21.955570 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87841754-1b3c-4126-a0fd-6ac0a5ba183f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:21.957224 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87841754-1b3c-4126-a0fd-6ac0a5ba183f None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:21.962807 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 385/763] 10.222.0.42 () {56 vars in 1112 bytes} [Sun Aug 28 00:42:21 2022] GET /identity/v3/services?type=network => generated 350 bytes in 292 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:21.968374 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87841754-1b3c-4126-a0fd-6ac0a5ba183f None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:21.971229 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f7265b0-5c0a-4b97-9255-b966ec56c9d4 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:22.019185 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54a29d4e-5ab5-4fd7-b776-94634996c710 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:22.019802 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a29d4e-5ab5-4fd7-b776-94634996c710 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:22.020081 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a29d4e-5ab5-4fd7-b776-94634996c710 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:22.020354 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a29d4e-5ab5-4fd7-b776-94634996c710 None admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:22.021316 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54a29d4e-5ab5-4fd7-b776-94634996c710 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:22.021618 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54a29d4e-5ab5-4fd7-b776-94634996c710 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:22.023354 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54a29d4e-5ab5-4fd7-b776-94634996c710 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:22.039604 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 379/764] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Aug 28 00:42:21 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:22.039604 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54a29d4e-5ab5-4fd7-b776-94634996c710 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:22.087576 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87841754-1b3c-4126-a0fd-6ac0a5ba183f None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:22.140233 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a6714f5-ebf3-40ed-9a5f-cb455c57c1ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:22.141265 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a6714f5-ebf3-40ed-9a5f-cb455c57c1ee None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:22.141761 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a6714f5-ebf3-40ed-9a5f-cb455c57c1ee None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:22.142209 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a6714f5-ebf3-40ed-9a5f-cb455c57c1ee None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:22.143153 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a6714f5-ebf3-40ed-9a5f-cb455c57c1ee None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:22.144812 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a6714f5-ebf3-40ed-9a5f-cb455c57c1ee None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:22.154739 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-2a6714f5-ebf3-40ed-9a5f-cb455c57c1ee None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 28 00:42:22.154739 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:22.154739 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 28 00:42:22.154739 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 28 00:42:22.154739 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 28 00:42:22.154739 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 28 00:42:22.154739 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 28 00:42:22.154739 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 28 00:42:22.154739 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 28 00:42:22.154739 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:22.154739 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 28 00:42:22.154739 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:22.154739 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:22.154739 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:22.154739 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:22.154739 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:22.154739 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:22.155799 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:22.155799 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:22.155799 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:22.155799 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:22.155799 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:22.155799 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:22.155799 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 28 00:42:22.155799 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 28 00:42:22.155799 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 28 00:42:22.155799 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 28 00:42:22.155799 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:22.155799 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:22.155799 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:22.155799 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:22.155799 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:22.155799 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:22.156696 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:22.156696 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:22.156696 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:22.156696 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:22.156696 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:22.156696 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:22.156696 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:22.156696 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:22.156696 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:22.156696 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:22.156696 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 28 00:42:22.156696 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 28 00:42:22.156696 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 28 00:42:22.156696 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:22.159403 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 386/765] 10.222.0.42 () {56 vars in 1129 bytes} [Sun Aug 28 00:42:21 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:22.160367 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a6714f5-ebf3-40ed-9a5f-cb455c57c1ee None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:22.166376 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54a29d4e-5ab5-4fd7-b776-94634996c710 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:22.168166 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:22.197904 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:22.215609 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5579884-de77-4049-bcc0-c36466bd0319 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:22.216341 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5579884-de77-4049-bcc0-c36466bd0319 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:22.217158 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5579884-de77-4049-bcc0-c36466bd0319 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:22.217606 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5579884-de77-4049-bcc0-c36466bd0319 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:22.218513 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5579884-de77-4049-bcc0-c36466bd0319 None admin] RBAC: Adding query filter params (interface=public, service_id=0616eb7e32cb42b0900e3ceab13361d8, region_id=RegionOne) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:22.219010 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5579884-de77-4049-bcc0-c36466bd0319 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:22.220729 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5579884-de77-4049-bcc0-c36466bd0319 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:22.236125 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 380/766] 10.222.0.42 () {56 vars in 1251 bytes} [Sun Aug 28 00:42:22 2022] GET /identity/v3/endpoints?service_id=0616eb7e32cb42b0900e3ceab13361d8&interface=public®ion_id=RegionOne => generated 194 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:22.236125 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5579884-de77-4049-bcc0-c36466bd0319 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:22.289546 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a6714f5-ebf3-40ed-9a5f-cb455c57c1ee None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:22.339298 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9f82027-8482-43dd-bb41-20bc534920a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:22.340106 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f82027-8482-43dd-bb41-20bc534920a1 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:22.340449 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f82027-8482-43dd-bb41-20bc534920a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:22.340789 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f82027-8482-43dd-bb41-20bc534920a1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:22.354674 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5579884-de77-4049-bcc0-c36466bd0319 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:22.356102 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9f82027-8482-43dd-bb41-20bc534920a1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:22.358947 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9f82027-8482-43dd-bb41-20bc534920a1 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:22.378049 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-f9f82027-8482-43dd-bb41-20bc534920a1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:42:22.378049 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:22.378049 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:22.378049 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:22.378049 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:22.378049 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:22.378049 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:22.378049 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:22.378049 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:22.378049 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:22.378049 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:22.378049 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:22.378049 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 28 00:42:22.378049 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 28 00:42:22.378049 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 28 00:42:22.378049 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 28 00:42:22.379071 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:22.379071 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:22.379071 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:22.379071 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:22.379071 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:22.379071 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:22.379071 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:22.379071 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:22.379071 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:22.379071 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:22.379071 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:22.379071 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:22.379071 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:22.379071 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:22.379071 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:22.379071 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:22.379071 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 28 00:42:22.380011 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 28 00:42:22.380011 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:42:22.380011 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:22.380011 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 387/767] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:42:22 2022] GET /identity/v3/domains/Default => generated 87 bytes in 219 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:22.385103 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9f82027-8482-43dd-bb41-20bc534920a1 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:22.401965 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8eebc72-9242-4e80-983d-885f2b2fb3b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:22.402578 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8eebc72-9242-4e80-983d-885f2b2fb3b2 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:22.402855 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8eebc72-9242-4e80-983d-885f2b2fb3b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:22.403531 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8eebc72-9242-4e80-983d-885f2b2fb3b2 None admin] PATH_INFO: `/v3/services` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:22.404286 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8eebc72-9242-4e80-983d-885f2b2fb3b2 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:22.404727 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8eebc72-9242-4e80-983d-885f2b2fb3b2 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:22.406746 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8eebc72-9242-4e80-983d-885f2b2fb3b2 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:22.417258 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 381/768] 10.222.0.42 () {56 vars in 1124 bytes} [Sun Aug 28 00:42:22 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:22.417258 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee7d0c98-9e20-4bd6-8797-edc2d85e1d93 None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:22.417258 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee7d0c98-9e20-4bd6-8797-edc2d85e1d93 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:22.417258 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee7d0c98-9e20-4bd6-8797-edc2d85e1d93 None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:22.417654 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 382/769] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:42:22 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:42:22.418697 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee7d0c98-9e20-4bd6-8797-edc2d85e1d93 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:22.450968 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:22.471444 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:22.490018 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9f82027-8482-43dd-bb41-20bc534920a1 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:22.524838 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-011fca65-8ee2-4d8d-9f1a-d986bfe2562a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:22.525493 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-011fca65-8ee2-4d8d-9f1a-d986bfe2562a None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:22.525765 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-011fca65-8ee2-4d8d-9f1a-d986bfe2562a None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:22.526037 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-011fca65-8ee2-4d8d-9f1a-d986bfe2562a None admin] PATH_INFO: `/v3/services` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:22.526667 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-011fca65-8ee2-4d8d-9f1a-d986bfe2562a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:22.528041 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-011fca65-8ee2-4d8d-9f1a-d986bfe2562a None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:22.532979 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee7d0c98-9e20-4bd6-8797-edc2d85e1d93 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:22.533700 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 388/770] 10.222.0.42 () {56 vars in 1087 bytes} [Sun Aug 28 00:42:22 2022] GET /identity/v3/services => generated 772 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:22.534486 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-011fca65-8ee2-4d8d-9f1a-d986bfe2562a None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:22.571356 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5a968a2-3d22-4c18-81ed-14ba80adb247 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:22.572400 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5a968a2-3d22-4c18-81ed-14ba80adb247 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:22.572853 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5a968a2-3d22-4c18-81ed-14ba80adb247 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:22.573305 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5a968a2-3d22-4c18-81ed-14ba80adb247 None admin] PATH_INFO: `/v3/domains` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:22.574163 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5a968a2-3d22-4c18-81ed-14ba80adb247 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:22.574462 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5a968a2-3d22-4c18-81ed-14ba80adb247 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:22.575898 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5a968a2-3d22-4c18-81ed-14ba80adb247 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:22.589528 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 383/771] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Aug 28 00:42:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:22.590962 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e73d479-0d01-42a6-900b-24ace9bbf027 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:22.591252 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e73d479-0d01-42a6-900b-24ace9bbf027 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:22.591521 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e73d479-0d01-42a6-900b-24ace9bbf027 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:22.610322 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-8e73d479-0d01-42a6-900b-24ace9bbf027 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:42:22.611560 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-8e73d479-0d01-42a6-900b-24ace9bbf027 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:42:22.679371 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-011fca65-8ee2-4d8d-9f1a-d986bfe2562a None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:22.695280 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 384/772] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:42:22 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 28 00:42:22.696432 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e73d479-0d01-42a6-900b-24ace9bbf027 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:22.725367 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:22.727416 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce9b7b55-d838-4d18-bc54-dd5e7b6b0f4e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:22.728372 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce9b7b55-d838-4d18-bc54-dd5e7b6b0f4e None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:22.728833 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce9b7b55-d838-4d18-bc54-dd5e7b6b0f4e None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:22.729158 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce9b7b55-d838-4d18-bc54-dd5e7b6b0f4e None admin] PATH_INFO: `/v3/services` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:22.729793 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce9b7b55-d838-4d18-bc54-dd5e7b6b0f4e None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:22.730093 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce9b7b55-d838-4d18-bc54-dd5e7b6b0f4e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:22.731757 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce9b7b55-d838-4d18-bc54-dd5e7b6b0f4e None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:22.737517 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 389/773] 10.222.0.42 () {56 vars in 1124 bytes} [Sun Aug 28 00:42:22 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:22.738462 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7821964-a527-47c7-9837-dd0f3b4e2c8b None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:22.738757 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7821964-a527-47c7-9837-dd0f3b4e2c8b None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:22.739025 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7821964-a527-47c7-9837-dd0f3b4e2c8b None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:22.740439 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 390/774] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:42:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:42:22.744077 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7821964-a527-47c7-9837-dd0f3b4e2c8b None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:22.760126 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:22.825900 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e73d479-0d01-42a6-900b-24ace9bbf027 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:22.861841 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7821964-a527-47c7-9837-dd0f3b4e2c8b None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:22.877199 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-852b9677-3de2-4863-b126-be0cef097431 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:22.878608 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-852b9677-3de2-4863-b126-be0cef097431 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:22.878894 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-852b9677-3de2-4863-b126-be0cef097431 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:22.879167 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-852b9677-3de2-4863-b126-be0cef097431 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:22.895801 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-852b9677-3de2-4863-b126-be0cef097431 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:22.899341 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-852b9677-3de2-4863-b126-be0cef097431 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:22.907576 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69af549c-9039-4576-954c-85ec2e9ea122 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:22.908770 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69af549c-9039-4576-954c-85ec2e9ea122 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:22.909328 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69af549c-9039-4576-954c-85ec2e9ea122 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:22.909818 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69af549c-9039-4576-954c-85ec2e9ea122 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:22.910916 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69af549c-9039-4576-954c-85ec2e9ea122 None admin] RBAC: Adding query filter params (interface=public, service_id=a057770f51084403af608c62ffb80bc9, region_id=RegionOne) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:22.911424 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69af549c-9039-4576-954c-85ec2e9ea122 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:22.913101 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69af549c-9039-4576-954c-85ec2e9ea122 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:22.915935 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-852b9677-3de2-4863-b126-be0cef097431 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 28 00:42:22.915935 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:22.915935 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:22.915935 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:22.915935 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:22.915935 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:22.915935 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:22.915935 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:22.915935 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:22.915935 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:22.915935 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:22.915935 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:22.915935 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 28 00:42:22.915935 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 28 00:42:22.915935 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 28 00:42:22.915935 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 28 00:42:22.916868 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:22.916868 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:22.916868 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 28 00:42:22.916868 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 28 00:42:22.916868 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 28 00:42:22.916868 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 28 00:42:22.916868 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:22.916868 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:22.916868 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:22.916868 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:22.916868 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:22.916868 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:22.916868 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:22.916868 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:22.916868 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:22.916868 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:22.916868 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:22.916868 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:22.917861 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:22.917861 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:22.917861 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 28 00:42:22.917861 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 28 00:42:22.917861 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 28 00:42:22.917861 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 28 00:42:22.917861 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 28 00:42:22.917861 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 28 00:42:22.917861 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 28 00:42:22.917861 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:42:22.922306 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 385/775] 10.222.0.42 () {56 vars in 1093 bytes} [Sun Aug 28 00:42:22 2022] GET /identity/v3/users/nova => generated 82 bytes in 226 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:22.923982 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a785bab-43f5-4aeb-9573-1f49143a02cb None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:22.924297 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a785bab-43f5-4aeb-9573-1f49143a02cb None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:22.924576 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a785bab-43f5-4aeb-9573-1f49143a02cb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:22.926291 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 391/776] 10.222.0.42 () {56 vars in 1251 bytes} [Sun Aug 28 00:42:22 2022] GET /identity/v3/endpoints?service_id=a057770f51084403af608c62ffb80bc9&interface=public®ion_id=RegionOne => generated 194 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 28 00:42:22.927712 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69af549c-9039-4576-954c-85ec2e9ea122 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:22.947392 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-9a785bab-43f5-4aeb-9573-1f49143a02cb None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:42:22.948918 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-9a785bab-43f5-4aeb-9573-1f49143a02cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:42:23.018724 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 386/777] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:42:22 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 28 00:42:23.020872 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:23.021104 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a785bab-43f5-4aeb-9573-1f49143a02cb None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:23.041488 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:23.057442 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69af549c-9039-4576-954c-85ec2e9ea122 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:23.106112 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14fd18bd-9325-463b-9580-bc21e5c2d696 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:23.107654 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14fd18bd-9325-463b-9580-bc21e5c2d696 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:23.108418 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14fd18bd-9325-463b-9580-bc21e5c2d696 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:23.108989 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14fd18bd-9325-463b-9580-bc21e5c2d696 None admin] PATH_INFO: `/v3/users` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:23.109935 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14fd18bd-9325-463b-9580-bc21e5c2d696 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:23.110444 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14fd18bd-9325-463b-9580-bc21e5c2d696 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:23.112109 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14fd18bd-9325-463b-9580-bc21e5c2d696 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:23.136793 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 392/778] 10.222.0.42 () {56 vars in 1133 bytes} [Sun Aug 28 00:42:22 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:23.137956 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14fd18bd-9325-463b-9580-bc21e5c2d696 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:23.155827 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a785bab-43f5-4aeb-9573-1f49143a02cb None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:23.196143 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-498553b5-e049-4318-a2e9-db85a7042da8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:23.197308 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-498553b5-e049-4318-a2e9-db85a7042da8 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:23.197852 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-498553b5-e049-4318-a2e9-db85a7042da8 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:23.198531 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-498553b5-e049-4318-a2e9-db85a7042da8 None admin] PATH_INFO: `/v3/services` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:23.199683 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-498553b5-e049-4318-a2e9-db85a7042da8 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:23.201281 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-498553b5-e049-4318-a2e9-db85a7042da8 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:23.209910 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 387/779] 10.222.0.42 () {56 vars in 1087 bytes} [Sun Aug 28 00:42:23 2022] GET /identity/v3/services => generated 772 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:23.211280 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-498553b5-e049-4318-a2e9-db85a7042da8 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:23.255321 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14fd18bd-9325-463b-9580-bc21e5c2d696 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:23.306689 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:23.314968 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:23.325629 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-762151be-0b6c-419c-b10d-4846e5c82afb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:23.326381 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-762151be-0b6c-419c-b10d-4846e5c82afb None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:23.326816 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-762151be-0b6c-419c-b10d-4846e5c82afb None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:23.327273 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-762151be-0b6c-419c-b10d-4846e5c82afb None admin] PATH_INFO: `/v3/roles/service` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:23.345890 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-498553b5-e049-4318-a2e9-db85a7042da8 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:23.347421 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-762151be-0b6c-419c-b10d-4846e5c82afb None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 28 00:42:23.352589 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-762151be-0b6c-419c-b10d-4846e5c82afb None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 28 00:42:23.352589 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:23.352589 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:23.352589 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:23.352589 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:23.352589 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:23.352589 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:23.352589 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:23.352589 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:23.352589 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:23.352589 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:23.352589 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:23.352589 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 28 00:42:23.352589 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 28 00:42:23.352589 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 28 00:42:23.352589 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 28 00:42:23.353626 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 28 00:42:23.353626 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 28 00:42:23.353626 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 28 00:42:23.353626 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 28 00:42:23.353626 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:23.353626 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:23.353626 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:23.353626 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:23.353626 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:23.353626 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:23.353626 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:23.353626 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:23.353626 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:23.353626 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:23.353626 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:23.353626 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:23.354498 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:23.354498 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:23.354498 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:23.354498 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:23.354498 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 28 00:42:23.354498 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 28 00:42:23.354498 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 28 00:42:23.354498 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 28 00:42:23.354498 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 28 00:42:23.354498 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 28 00:42:23.354498 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 28 00:42:23.354498 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:23.358976 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 393/780] 10.222.0.42 () {56 vars in 1102 bytes} [Sun Aug 28 00:42:23 2022] GET /identity/v3/roles/service => generated 85 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:23.359983 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-762151be-0b6c-419c-b10d-4846e5c82afb None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:23.394338 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2923a7b5-7440-440f-92c2-f56a1109cb42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:23.395994 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2923a7b5-7440-440f-92c2-f56a1109cb42 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:23.396462 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2923a7b5-7440-440f-92c2-f56a1109cb42 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:23.396878 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2923a7b5-7440-440f-92c2-f56a1109cb42 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:23.411739 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2923a7b5-7440-440f-92c2-f56a1109cb42 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:23.413411 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2923a7b5-7440-440f-92c2-f56a1109cb42 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:23.430384 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-2923a7b5-7440-440f-92c2-f56a1109cb42 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:42:23.430384 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:23.430384 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:23.430384 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:23.430384 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:23.430384 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:23.430384 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:23.430384 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:23.430384 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:23.430384 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:23.430384 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:23.430384 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:23.430384 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 28 00:42:23.430384 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 28 00:42:23.430384 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 28 00:42:23.430384 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 28 00:42:23.431377 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:23.431377 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:23.431377 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:23.431377 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:23.431377 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:23.431377 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:23.431377 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:23.431377 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:23.431377 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:23.431377 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:23.431377 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:23.431377 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:23.431377 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:23.431377 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:23.431377 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:23.431377 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:23.431377 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 28 00:42:23.432354 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 28 00:42:23.432354 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:42:23.432354 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:42:23.432354 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 388/781] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:42:23 2022] GET /identity/v3/domains/Default => generated 87 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:23.432354 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2923a7b5-7440-440f-92c2-f56a1109cb42 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:23.485030 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-762151be-0b6c-419c-b10d-4846e5c82afb None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:23.531244 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4be31be3-de50-4eda-ad05-6c05e6296b22 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:23.532251 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4be31be3-de50-4eda-ad05-6c05e6296b22 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:23.533168 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4be31be3-de50-4eda-ad05-6c05e6296b22 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:23.533649 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4be31be3-de50-4eda-ad05-6c05e6296b22 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:23.548570 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4be31be3-de50-4eda-ad05-6c05e6296b22 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 28 00:42:23.555922 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-4be31be3-de50-4eda-ad05-6c05e6296b22 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 28 00:42:23.555922 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:23.555922 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:23.555922 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:23.555922 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:23.555922 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:23.555922 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:23.555922 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:23.555922 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:23.555922 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:23.555922 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:23.555922 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:23.555922 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 28 00:42:23.555922 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 28 00:42:23.555922 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 28 00:42:23.555922 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 28 00:42:23.557090 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 28 00:42:23.557090 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 28 00:42:23.557090 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 28 00:42:23.557090 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 28 00:42:23.557090 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:23.557090 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:23.557090 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:23.557090 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:23.557090 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:23.557090 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:23.557090 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:23.557090 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:23.557090 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:23.557090 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:23.557090 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:23.557090 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:23.558158 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:23.558158 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:23.558158 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:23.558158 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:23.558158 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 28 00:42:23.558158 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 28 00:42:23.558158 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 28 00:42:23.558158 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 28 00:42:23.558158 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 28 00:42:23.558158 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 28 00:42:23.558158 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 28 00:42:23.558158 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:23.559576 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2923a7b5-7440-440f-92c2-f56a1109cb42 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:23.560728 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 394/782] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Aug 28 00:42:23 2022] GET /identity/v3/roles/admin => generated 83 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:23.562067 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4be31be3-de50-4eda-ad05-6c05e6296b22 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:23.587854 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:23.598866 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dbaf637-2c77-4b52-a047-c309ad741dd4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:23.599568 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dbaf637-2c77-4b52-a047-c309ad741dd4 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:23.599795 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dbaf637-2c77-4b52-a047-c309ad741dd4 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:23.600064 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dbaf637-2c77-4b52-a047-c309ad741dd4 None admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:23.600774 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dbaf637-2c77-4b52-a047-c309ad741dd4 None admin] RBAC: Adding query filter params (name=service) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:23.601149 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dbaf637-2c77-4b52-a047-c309ad741dd4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:23.602633 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dbaf637-2c77-4b52-a047-c309ad741dd4 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:23.614504 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 389/783] 10.222.0.42 () {56 vars in 1103 bytes} [Sun Aug 28 00:42:23 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:23.615252 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dbaf637-2c77-4b52-a047-c309ad741dd4 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:23.639147 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:23.697080 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4be31be3-de50-4eda-ad05-6c05e6296b22 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:23.735546 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb365b8f-f86c-433e-9b75-1879caadc1d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:23.736506 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb365b8f-f86c-433e-9b75-1879caadc1d6 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:23.736999 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb365b8f-f86c-433e-9b75-1879caadc1d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:23.737486 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb365b8f-f86c-433e-9b75-1879caadc1d6 None admin] PATH_INFO: `/v3/domains` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:23.738311 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb365b8f-f86c-433e-9b75-1879caadc1d6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:23.738799 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb365b8f-f86c-433e-9b75-1879caadc1d6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:23.740776 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb365b8f-f86c-433e-9b75-1879caadc1d6 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:23.756262 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 395/784] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Aug 28 00:42:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:23.757702 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb365b8f-f86c-433e-9b75-1879caadc1d6 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:23.768818 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dbaf637-2c77-4b52-a047-c309ad741dd4 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:23.811667 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dadb53d1-97aa-4013-8073-e2c8d6de750f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:23.812744 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dadb53d1-97aa-4013-8073-e2c8d6de750f None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:23.813100 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dadb53d1-97aa-4013-8073-e2c8d6de750f None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:23.813433 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dadb53d1-97aa-4013-8073-e2c8d6de750f None admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:23.814430 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dadb53d1-97aa-4013-8073-e2c8d6de750f None admin] RBAC: Adding query filter params (name=admin) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:23.814794 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dadb53d1-97aa-4013-8073-e2c8d6de750f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:23.816492 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dadb53d1-97aa-4013-8073-e2c8d6de750f None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:23.826497 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 390/785] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Aug 28 00:42:23 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:23.828074 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dadb53d1-97aa-4013-8073-e2c8d6de750f None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:23.880606 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:23.889220 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb365b8f-f86c-433e-9b75-1879caadc1d6 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:23.907001 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:23.933004 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dadb53d1-97aa-4013-8073-e2c8d6de750f None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:23.935509 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4651641d-30d2-48a9-8014-8df333e7cb26 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:23.936806 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4651641d-30d2-48a9-8014-8df333e7cb26 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:23.937306 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4651641d-30d2-48a9-8014-8df333e7cb26 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:23.937767 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4651641d-30d2-48a9-8014-8df333e7cb26 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:23.950231 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4651641d-30d2-48a9-8014-8df333e7cb26 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:23.952192 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4651641d-30d2-48a9-8014-8df333e7cb26 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:23.968318 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-168cc27e-5859-4974-98c5-e34a68616b46 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:23.968637 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-4651641d-30d2-48a9-8014-8df333e7cb26 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:42:23.968637 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:23.968637 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:23.968637 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:23.968637 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:23.968637 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:23.968637 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:23.968637 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:23.968637 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:23.968637 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:23.968637 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:23.968637 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:23.968637 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 28 00:42:23.968637 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 28 00:42:23.968637 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 28 00:42:23.968637 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 28 00:42:23.969584 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:23.969584 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:23.969584 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:23.969584 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:23.969584 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:23.969584 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:23.969584 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:23.969584 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:23.969584 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:23.969584 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:23.969584 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:23.969584 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:23.969584 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:23.969584 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:23.969584 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:23.969584 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:23.969584 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 28 00:42:23.970472 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 28 00:42:23.970472 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:42:23.970472 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:23.970650 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-168cc27e-5859-4974-98c5-e34a68616b46 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:23.970650 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-168cc27e-5859-4974-98c5-e34a68616b46 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:23.970650 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-168cc27e-5859-4974-98c5-e34a68616b46 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:23.977160 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-168cc27e-5859-4974-98c5-e34a68616b46 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:23.977425 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 396/786] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:42:23 2022] GET /identity/v3/domains/Default => generated 87 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:23.978488 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4651641d-30d2-48a9-8014-8df333e7cb26 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:23.978743 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-168cc27e-5859-4974-98c5-e34a68616b46 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:23.990911 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-168cc27e-5859-4974-98c5-e34a68616b46 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 28 00:42:23.990911 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:23.990911 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:23.990911 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:23.990911 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:23.990911 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:23.990911 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:23.990911 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:23.990911 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:23.990911 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:23.990911 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:23.990911 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:23.990911 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 28 00:42:23.990911 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 28 00:42:23.990911 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 28 00:42:23.991788 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 28 00:42:23.991788 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:23.991788 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:23.991788 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:23.991788 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:23.991788 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:23.991788 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:23.991788 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:23.991788 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:23.991788 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:23.991788 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:23.991788 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:23.991788 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:23.991788 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:23.991788 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:23.991788 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:23.991788 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:23.992705 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 28 00:42:23.992705 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 28 00:42:23.992705 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 28 00:42:23.992705 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 28 00:42:23.992705 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 28 00:42:23.992705 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 28 00:42:23.992705 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 28 00:42:23.992705 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:42:23.992705 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 391/787] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Aug 28 00:42:23 2022] GET /identity/v3/projects/service => generated 88 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:23.993230 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-168cc27e-5859-4974-98c5-e34a68616b46 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:24.095861 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-168cc27e-5859-4974-98c5-e34a68616b46 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:24.099618 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4651641d-30d2-48a9-8014-8df333e7cb26 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:24.107004 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:24.140812 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c5045af-32e7-4984-aeb2-aafe2e9d5787 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:24.141458 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c5045af-32e7-4984-aeb2-aafe2e9d5787 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:24.141737 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c5045af-32e7-4984-aeb2-aafe2e9d5787 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:24.142006 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c5045af-32e7-4984-aeb2-aafe2e9d5787 None admin] PATH_INFO: `/v3/domains` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:24.142654 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c5045af-32e7-4984-aeb2-aafe2e9d5787 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:24.142953 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c5045af-32e7-4984-aeb2-aafe2e9d5787 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:24.144324 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c5045af-32e7-4984-aeb2-aafe2e9d5787 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:24.146531 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b3ae4ba-88e1-4112-aaa4-f00797102f79 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:24.147631 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b3ae4ba-88e1-4112-aaa4-f00797102f79 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:24.148231 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b3ae4ba-88e1-4112-aaa4-f00797102f79 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:24.148696 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b3ae4ba-88e1-4112-aaa4-f00797102f79 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:24.162011 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 392/788] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Aug 28 00:42:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:24.162011 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c5045af-32e7-4984-aeb2-aafe2e9d5787 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:24.165516 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b3ae4ba-88e1-4112-aaa4-f00797102f79 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:24.167341 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b3ae4ba-88e1-4112-aaa4-f00797102f79 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:24.183038 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-3b3ae4ba-88e1-4112-aaa4-f00797102f79 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:42:24.183038 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:24.183038 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:24.183038 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:24.183038 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:24.183038 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:24.183038 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:24.183038 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:24.183038 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:24.183038 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:24.183038 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:24.183038 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:24.183038 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 28 00:42:24.183038 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 28 00:42:24.183038 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 28 00:42:24.183038 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 28 00:42:24.184006 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:24.184006 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:24.184006 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:24.184006 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:24.184006 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:24.184006 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:24.184006 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:24.184006 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:24.184006 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:24.184006 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:24.184006 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:24.184006 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:24.184006 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:24.184006 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:24.184006 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:24.184006 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:24.184006 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 28 00:42:24.185006 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 28 00:42:24.185006 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:42:24.185006 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:24.187118 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 397/789] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:42:23 2022] GET /identity/v3/domains/Default => generated 87 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:24.188144 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b3ae4ba-88e1-4112-aaa4-f00797102f79 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:24.209047 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:24.260775 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c5045af-32e7-4984-aeb2-aafe2e9d5787 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:24.296714 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4305f5eb-3f87-4bc6-869f-63d2d591f5d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:24.297737 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4305f5eb-3f87-4bc6-869f-63d2d591f5d8 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:24.297737 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4305f5eb-3f87-4bc6-869f-63d2d591f5d8 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:24.298017 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4305f5eb-3f87-4bc6-869f-63d2d591f5d8 None admin] PATH_INFO: `/v3/projects` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:24.298729 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4305f5eb-3f87-4bc6-869f-63d2d591f5d8 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:24.299033 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4305f5eb-3f87-4bc6-869f-63d2d591f5d8 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:24.300495 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4305f5eb-3f87-4bc6-869f-63d2d591f5d8 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:24.314390 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b3ae4ba-88e1-4112-aaa4-f00797102f79 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:24.316226 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 393/790] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Aug 28 00:42:24 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:24.318541 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4305f5eb-3f87-4bc6-869f-63d2d591f5d8 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:24.353228 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a95b209-0573-43e0-af56-865db8347951 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:24.353895 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a95b209-0573-43e0-af56-865db8347951 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:24.354173 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a95b209-0573-43e0-af56-865db8347951 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:24.354446 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a95b209-0573-43e0-af56-865db8347951 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:24.362485 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:24.363922 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a95b209-0573-43e0-af56-865db8347951 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:24.365470 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a95b209-0573-43e0-af56-865db8347951 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:24.378964 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-6a95b209-0573-43e0-af56-865db8347951 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 28 00:42:24.378964 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:24.378964 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:24.378964 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:24.378964 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:24.378964 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:24.378964 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:24.378964 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:24.378964 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:24.378964 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:24.378964 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:24.378964 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:24.378964 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 28 00:42:24.378964 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 28 00:42:24.378964 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 28 00:42:24.378964 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 28 00:42:24.379949 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:24.379949 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:24.379949 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 28 00:42:24.379949 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 28 00:42:24.379949 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 28 00:42:24.379949 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 28 00:42:24.379949 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:24.379949 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:24.379949 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:24.379949 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:24.379949 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:24.379949 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:24.379949 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:24.379949 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:24.379949 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:24.379949 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:24.379949 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:24.379949 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:24.380883 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:24.380883 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:24.380883 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 28 00:42:24.380883 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 28 00:42:24.380883 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 28 00:42:24.380883 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 28 00:42:24.380883 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 28 00:42:24.380883 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 28 00:42:24.380883 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 28 00:42:24.380883 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:24.384838 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 398/791] 10.222.0.42 () {56 vars in 1117 bytes} [Sun Aug 28 00:42:24 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:24.385728 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a95b209-0573-43e0-af56-865db8347951 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:24.448629 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4305f5eb-3f87-4bc6-869f-63d2d591f5d8 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:24.487064 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:24.487704 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d97f064-762d-4874-88bc-642cc776b3ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:24.488600 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d97f064-762d-4874-88bc-642cc776b3ca None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:24.489065 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d97f064-762d-4874-88bc-642cc776b3ca None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:24.489626 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d97f064-762d-4874-88bc-642cc776b3ca None admin] PATH_INFO: `/v3/domains` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:24.491238 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d97f064-762d-4874-88bc-642cc776b3ca None admin] RBAC: Adding query filter params (name=Default) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:24.491688 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d97f064-762d-4874-88bc-642cc776b3ca None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:24.493236 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d97f064-762d-4874-88bc-642cc776b3ca None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:24.507241 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 394/792] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Aug 28 00:42:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:24.508417 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d97f064-762d-4874-88bc-642cc776b3ca None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:24.518052 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a95b209-0573-43e0-af56-865db8347951 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:24.556454 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e0e9e2d-b6c7-4a86-9bcf-8c9bb1d4c96b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:24.557282 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e0e9e2d-b6c7-4a86-9bcf-8c9bb1d4c96b None admin] REQUEST_METHOD: `PUT` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:24.557923 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e0e9e2d-b6c7-4a86-9bcf-8c9bb1d4c96b None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:24.558196 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e0e9e2d-b6c7-4a86-9bcf-8c9bb1d4c96b None admin] PATH_INFO: `/v3/projects/f53e6fdb369b480fbebb4086a505df67/users/168b07df50c24593ba73dbde5b037aed/roles/f08d6043261747e3b4763c744eef7dfb` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:24.560301 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e0e9e2d-b6c7-4a86-9bcf-8c9bb1d4c96b None admin] RBAC: Authorizing `identity:create_grant(project_id=f53e6fdb369b480fbebb4086a505df67, user_id=168b07df50c24593ba73dbde5b037aed, role_id=f08d6043261747e3b4763c744eef7dfb)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:24.561919 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e0e9e2d-b6c7-4a86-9bcf-8c9bb1d4c96b None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:24.581111 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 399/793] 10.222.0.42 () {58 vars in 1439 bytes} [Sun Aug 28 00:42:24 2022] PUT /identity/v3/projects/f53e6fdb369b480fbebb4086a505df67/users/168b07df50c24593ba73dbde5b037aed/roles/f08d6043261747e3b4763c744eef7dfb => generated 0 bytes in 191 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:42:24.581111 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e0e9e2d-b6c7-4a86-9bcf-8c9bb1d4c96b None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:24.617549 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:24.629163 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d97f064-762d-4874-88bc-642cc776b3ca None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:24.670390 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87af3263-db8c-4856-9cae-0c55831aed54 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:24.671161 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87af3263-db8c-4856-9cae-0c55831aed54 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:24.671573 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87af3263-db8c-4856-9cae-0c55831aed54 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:24.671852 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87af3263-db8c-4856-9cae-0c55831aed54 None admin] PATH_INFO: `/v3/users` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:24.672661 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87af3263-db8c-4856-9cae-0c55831aed54 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:24.673089 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87af3263-db8c-4856-9cae-0c55831aed54 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:24.674764 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87af3263-db8c-4856-9cae-0c55831aed54 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:24.701784 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 395/794] 10.222.0.42 () {56 vars in 1149 bytes} [Sun Aug 28 00:42:24 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:24.703306 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5db053b7-f7cc-4780-970b-74add87d2310 None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:24.703625 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5db053b7-f7cc-4780-970b-74add87d2310 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:24.704300 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5db053b7-f7cc-4780-970b-74add87d2310 None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:24.706713 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 396/795] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:42:24 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:42:24.707273 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5db053b7-f7cc-4780-970b-74add87d2310 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:24.737299 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e0e9e2d-b6c7-4a86-9bcf-8c9bb1d4c96b None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:24.781200 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1aa8cfa-5712-4e0d-b402-7c82aa86e5ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:24.782671 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1aa8cfa-5712-4e0d-b402-7c82aa86e5ee None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:24.783126 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1aa8cfa-5712-4e0d-b402-7c82aa86e5ee None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:24.783811 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1aa8cfa-5712-4e0d-b402-7c82aa86e5ee None admin] PATH_INFO: `/v3/users/swift` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:24.787872 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:24.798259 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1aa8cfa-5712-4e0d-b402-7c82aa86e5ee None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:24.799721 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1aa8cfa-5712-4e0d-b402-7c82aa86e5ee None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:24.816222 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-b1aa8cfa-5712-4e0d-b402-7c82aa86e5ee None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Aug 28 00:42:24.816222 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:24.816222 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:24.816222 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:24.816222 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:24.816222 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:24.816222 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:24.816222 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:24.816222 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:24.816222 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:24.816222 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:24.816222 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:24.816222 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 28 00:42:24.816222 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 28 00:42:24.816222 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 28 00:42:24.816222 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 28 00:42:24.817233 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:24.817233 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:24.817233 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 28 00:42:24.817233 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 28 00:42:24.817233 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 28 00:42:24.817233 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 28 00:42:24.817233 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:24.817233 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:24.817233 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:24.817233 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:24.817233 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:24.817233 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:24.817233 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:24.817233 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:24.817233 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:24.817233 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:24.817233 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:24.817233 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:24.818172 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:24.818172 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:24.818172 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 28 00:42:24.818172 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 28 00:42:24.818172 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 28 00:42:24.818172 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 28 00:42:24.818172 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 28 00:42:24.818172 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 28 00:42:24.818172 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Aug 28 00:42:24.818172 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:24.818709 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 400/796] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Aug 28 00:42:24 2022] GET /identity/v3/users/swift => generated 83 bytes in 242 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:24.819837 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8c72747-a109-4f46-98a0-593ec8638b3e None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:24.820119 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8c72747-a109-4f46-98a0-593ec8638b3e None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:24.820389 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8c72747-a109-4f46-98a0-593ec8638b3e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:24.839236 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-e8c72747-a109-4f46-98a0-593ec8638b3e None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:42:24.840543 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-e8c72747-a109-4f46-98a0-593ec8638b3e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:42:24.844789 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5db053b7-f7cc-4780-970b-74add87d2310 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:24.892180 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f80405a-3796-42e6-a118-a15f55c127fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:24.892820 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f80405a-3796-42e6-a118-a15f55c127fb None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:24.893162 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f80405a-3796-42e6-a118-a15f55c127fb None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:24.893438 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f80405a-3796-42e6-a118-a15f55c127fb None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:24.906177 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f80405a-3796-42e6-a118-a15f55c127fb None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:24.907653 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f80405a-3796-42e6-a118-a15f55c127fb None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:24.918258 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:24.930354 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 401/797] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:42:24 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 28 00:42:24.931518 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8c72747-a109-4f46-98a0-593ec8638b3e None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:24.933305 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-3f80405a-3796-42e6-a118-a15f55c127fb None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:42:24.933305 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:24.933305 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:24.933305 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:24.933305 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:24.933305 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:24.933305 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:24.933305 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:24.933305 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:24.933305 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:24.933305 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:24.933305 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:24.933305 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 28 00:42:24.933305 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 28 00:42:24.933305 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 28 00:42:24.933305 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 28 00:42:24.934250 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:24.934250 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:24.934250 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:24.934250 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:24.934250 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:24.934250 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:24.934250 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:24.934250 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:24.934250 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:24.934250 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:24.934250 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:24.934250 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:24.934250 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:24.934250 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:24.934250 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:24.934250 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:24.934250 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 28 00:42:24.935138 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 28 00:42:24.935138 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:42:24.935138 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:42:24.943634 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 397/798] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:42:24 2022] GET /identity/v3/domains/Default => generated 87 bytes in 231 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:24.947138 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f80405a-3796-42e6-a118-a15f55c127fb None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:25.052085 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8c72747-a109-4f46-98a0-593ec8638b3e None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:25.089522 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:25.094438 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-061f63b2-0c80-49e7-a144-b710f812894c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:25.095331 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-061f63b2-0c80-49e7-a144-b710f812894c None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:25.095931 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-061f63b2-0c80-49e7-a144-b710f812894c None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:25.096412 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-061f63b2-0c80-49e7-a144-b710f812894c None admin] PATH_INFO: `/v3/users` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:25.097547 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-061f63b2-0c80-49e7-a144-b710f812894c None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:25.098313 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-061f63b2-0c80-49e7-a144-b710f812894c None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:25.099936 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-061f63b2-0c80-49e7-a144-b710f812894c None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:25.120342 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 402/799] 10.222.0.42 () {56 vars in 1135 bytes} [Sun Aug 28 00:42:24 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:25.125545 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-061f63b2-0c80-49e7-a144-b710f812894c None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:25.156031 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f80405a-3796-42e6-a118-a15f55c127fb None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:25.233213 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-061f63b2-0c80-49e7-a144-b710f812894c None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:25.269724 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e61f7099-12e2-41b6-8f52-868cdef59a50 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:25.270768 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e61f7099-12e2-41b6-8f52-868cdef59a50 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:25.271292 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e61f7099-12e2-41b6-8f52-868cdef59a50 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:25.271780 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e61f7099-12e2-41b6-8f52-868cdef59a50 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:25.283309 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e61f7099-12e2-41b6-8f52-868cdef59a50 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:25.285237 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e61f7099-12e2-41b6-8f52-868cdef59a50 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:25.293982 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f5765e6-7c4b-4db8-95c2-6f5a2b751589 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:25.295059 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f5765e6-7c4b-4db8-95c2-6f5a2b751589 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:25.295395 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f5765e6-7c4b-4db8-95c2-6f5a2b751589 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:25.295800 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f5765e6-7c4b-4db8-95c2-6f5a2b751589 None admin] PATH_INFO: `/v3/domains` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:25.296758 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f5765e6-7c4b-4db8-95c2-6f5a2b751589 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:25.297294 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f5765e6-7c4b-4db8-95c2-6f5a2b751589 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:25.299479 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f5765e6-7c4b-4db8-95c2-6f5a2b751589 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:25.302989 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-e61f7099-12e2-41b6-8f52-868cdef59a50 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:42:25.302989 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:25.302989 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:25.302989 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:25.302989 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:25.302989 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:25.302989 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:25.302989 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:25.302989 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:25.302989 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:25.302989 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:25.302989 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:25.302989 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 28 00:42:25.302989 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 28 00:42:25.302989 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 28 00:42:25.302989 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 28 00:42:25.303906 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:25.303906 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:25.303906 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:25.303906 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:25.303906 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:25.303906 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:25.303906 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:25.303906 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:25.303906 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:25.303906 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:25.303906 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:25.303906 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:25.303906 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:25.303906 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:25.303906 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:25.303906 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:25.303906 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 28 00:42:25.304804 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 28 00:42:25.304804 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:42:25.304804 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:25.306863 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 403/800] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:42:25 2022] GET /identity/v3/domains/Default => generated 87 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 28 00:42:25.317017 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e61f7099-12e2-41b6-8f52-868cdef59a50 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:25.328593 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 398/801] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Aug 28 00:42:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 378 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:25.333227 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f5765e6-7c4b-4db8-95c2-6f5a2b751589 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:25.369449 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:25.424073 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:25.426409 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e61f7099-12e2-41b6-8f52-868cdef59a50 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:25.462592 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7122ec8-8685-4c6c-b405-a023c4c797d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:25.463194 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7122ec8-8685-4c6c-b405-a023c4c797d4 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:25.463472 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7122ec8-8685-4c6c-b405-a023c4c797d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:25.463742 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7122ec8-8685-4c6c-b405-a023c4c797d4 None admin] PATH_INFO: `/v3/domains` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:25.464376 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7122ec8-8685-4c6c-b405-a023c4c797d4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:25.464674 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7122ec8-8685-4c6c-b405-a023c4c797d4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:25.466078 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7122ec8-8685-4c6c-b405-a023c4c797d4 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:25.487076 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 404/802] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Aug 28 00:42:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:25.487076 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-739d78b1-9e5e-46ff-9b4e-007197cff064 None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:25.487076 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-739d78b1-9e5e-46ff-9b4e-007197cff064 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:25.487076 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-739d78b1-9e5e-46ff-9b4e-007197cff064 None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:25.487076 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 405/803] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:42:25 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:42:25.487076 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-739d78b1-9e5e-46ff-9b4e-007197cff064 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:25.487588 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f5765e6-7c4b-4db8-95c2-6f5a2b751589 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:25.535983 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71a1dd8d-c295-4e35-a4a7-72b0ef955325 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:25.536911 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71a1dd8d-c295-4e35-a4a7-72b0ef955325 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:25.537358 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71a1dd8d-c295-4e35-a4a7-72b0ef955325 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:25.537762 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71a1dd8d-c295-4e35-a4a7-72b0ef955325 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:25.548009 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71a1dd8d-c295-4e35-a4a7-72b0ef955325 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:25.550329 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71a1dd8d-c295-4e35-a4a7-72b0ef955325 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:25.568364 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-71a1dd8d-c295-4e35-a4a7-72b0ef955325 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 28 00:42:25.568364 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:25.568364 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:25.568364 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:25.568364 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:25.568364 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:25.568364 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:25.568364 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:25.568364 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:25.568364 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:25.568364 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:25.568364 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:25.568364 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 28 00:42:25.568364 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 28 00:42:25.568364 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 28 00:42:25.569329 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 28 00:42:25.569329 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:25.569329 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:25.569329 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:25.569329 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:25.569329 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:25.569329 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:25.569329 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:25.569329 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:25.569329 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:25.569329 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:25.569329 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:25.569329 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:25.569329 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:25.569329 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:25.569329 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:25.569329 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:25.570244 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 28 00:42:25.570244 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 28 00:42:25.570244 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 28 00:42:25.570244 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 28 00:42:25.570244 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 28 00:42:25.570244 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 28 00:42:25.570244 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 28 00:42:25.570244 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:42:25.573262 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 399/804] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Aug 28 00:42:25 2022] GET /identity/v3/projects/service => generated 88 bytes in 241 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:25.574437 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c71066c6-a6cd-4381-9da5-10f91436a136 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:25.574718 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c71066c6-a6cd-4381-9da5-10f91436a136 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:25.574984 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c71066c6-a6cd-4381-9da5-10f91436a136 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:25.594436 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-c71066c6-a6cd-4381-9da5-10f91436a136 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:42:25.594788 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:25.595636 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-c71066c6-a6cd-4381-9da5-10f91436a136 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:42:25.602988 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-739d78b1-9e5e-46ff-9b4e-007197cff064 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:25.642140 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa39e192-4143-4afa-aeb5-6a98cb77b3f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:25.642777 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa39e192-4143-4afa-aeb5-6a98cb77b3f5 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:25.643115 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa39e192-4143-4afa-aeb5-6a98cb77b3f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:25.643393 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa39e192-4143-4afa-aeb5-6a98cb77b3f5 None admin] PATH_INFO: `/v3/services/network` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:25.643988 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa39e192-4143-4afa-aeb5-6a98cb77b3f5 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:25.645410 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa39e192-4143-4afa-aeb5-6a98cb77b3f5 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:25.655240 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-fa39e192-4143-4afa-aeb5-6a98cb77b3f5 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Aug 28 00:42:25.655240 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:25.655240 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 28 00:42:25.655240 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 28 00:42:25.655240 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 28 00:42:25.655240 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 28 00:42:25.655240 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 28 00:42:25.655240 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 28 00:42:25.655240 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Aug 28 00:42:25.655240 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:25.655240 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 28 00:42:25.655240 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:25.655240 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:25.655240 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:25.655240 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:25.655240 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:25.655240 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:25.656378 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:25.656378 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:25.656378 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:25.656378 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:25.656378 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:25.656378 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:25.656378 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 28 00:42:25.656378 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 28 00:42:25.656378 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 28 00:42:25.656378 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 28 00:42:25.656378 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:25.656378 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:25.656378 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:25.656378 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:25.656378 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:25.656378 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:25.657249 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:25.657249 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:25.657249 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:25.657249 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:25.657249 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:25.657249 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:25.657249 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:25.657249 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:25.657249 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:25.657249 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:25.657249 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 28 00:42:25.657249 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 28 00:42:25.657249 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Aug 28 00:42:25.657249 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:25.661330 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 406/805] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Aug 28 00:42:25 2022] GET /identity/v3/services/network => generated 88 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:25.662187 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa39e192-4143-4afa-aeb5-6a98cb77b3f5 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:25.774355 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa39e192-4143-4afa-aeb5-6a98cb77b3f5 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:25.812280 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cbde872-9977-4a53-b427-dd68cce34d05 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:25.812924 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cbde872-9977-4a53-b427-dd68cce34d05 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:25.813233 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cbde872-9977-4a53-b427-dd68cce34d05 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:25.813504 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cbde872-9977-4a53-b427-dd68cce34d05 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:25.822312 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cbde872-9977-4a53-b427-dd68cce34d05 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:25.823972 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cbde872-9977-4a53-b427-dd68cce34d05 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:25.844772 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 400/806] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:42:25 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 262 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 28 00:42:25.844772 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c71066c6-a6cd-4381-9da5-10f91436a136 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:25.845379 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-6cbde872-9977-4a53-b427-dd68cce34d05 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 28 00:42:25.845379 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:25.845379 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:25.845379 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:25.845379 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:25.845379 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:25.845379 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:25.845379 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:25.845379 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:25.845379 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:25.845379 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:25.845379 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:25.845379 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 28 00:42:25.845379 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 28 00:42:25.845379 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 28 00:42:25.846189 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 28 00:42:25.846189 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:25.846189 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:25.846189 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:25.846189 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:25.846189 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:25.846189 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:25.846189 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:25.846189 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:25.846189 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:25.846189 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:25.846189 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:25.846189 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:25.846189 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:25.846189 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:25.846189 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:25.846189 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:25.847058 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 28 00:42:25.847058 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 28 00:42:25.847058 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 28 00:42:25.847058 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 28 00:42:25.847058 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 28 00:42:25.847058 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 28 00:42:25.847058 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 28 00:42:25.847058 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:25.847058 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 407/807] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Aug 28 00:42:25 2022] GET /identity/v3/projects/service => generated 88 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:25.847058 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cbde872-9977-4a53-b427-dd68cce34d05 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:25.872373 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:25.908228 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:25.967674 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c71066c6-a6cd-4381-9da5-10f91436a136 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:26.012453 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-041822ae-bcaf-42f7-92e2-f109199973e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:26.013104 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-041822ae-bcaf-42f7-92e2-f109199973e4 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:26.013445 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-041822ae-bcaf-42f7-92e2-f109199973e4 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:26.013675 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-041822ae-bcaf-42f7-92e2-f109199973e4 None admin] PATH_INFO: `/v3/projects` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:26.014358 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-041822ae-bcaf-42f7-92e2-f109199973e4 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:26.014661 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-041822ae-bcaf-42f7-92e2-f109199973e4 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:26.016118 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-041822ae-bcaf-42f7-92e2-f109199973e4 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:26.033910 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 401/808] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Aug 28 00:42:25 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:26.033910 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-041822ae-bcaf-42f7-92e2-f109199973e4 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:26.038793 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cbde872-9977-4a53-b427-dd68cce34d05 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:26.112362 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bedad59e-6342-4c40-8b0d-d339cc800fd4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:26.114319 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bedad59e-6342-4c40-8b0d-d339cc800fd4 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:26.116207 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bedad59e-6342-4c40-8b0d-d339cc800fd4 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:26.117253 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bedad59e-6342-4c40-8b0d-d339cc800fd4 None admin] PATH_INFO: `/v3/services` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:26.118547 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bedad59e-6342-4c40-8b0d-d339cc800fd4 None admin] RBAC: Adding query filter params (name=network) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:26.119009 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bedad59e-6342-4c40-8b0d-d339cc800fd4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:26.122577 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bedad59e-6342-4c40-8b0d-d339cc800fd4 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:26.133135 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 408/809] 10.222.0.42 () {56 vars in 1112 bytes} [Sun Aug 28 00:42:25 2022] GET /identity/v3/services?name=network => generated 124 bytes in 285 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:26.133135 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bedad59e-6342-4c40-8b0d-d339cc800fd4 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:26.153477 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-041822ae-bcaf-42f7-92e2-f109199973e4 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:26.165072 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:26.191573 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25f92598-c89e-46de-8a31-4acc1bde12c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:26.192478 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25f92598-c89e-46de-8a31-4acc1bde12c0 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:26.192931 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25f92598-c89e-46de-8a31-4acc1bde12c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:26.193374 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25f92598-c89e-46de-8a31-4acc1bde12c0 None admin] PATH_INFO: `/v3/projects` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:26.194235 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25f92598-c89e-46de-8a31-4acc1bde12c0 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:26.194696 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25f92598-c89e-46de-8a31-4acc1bde12c0 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:26.196287 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25f92598-c89e-46de-8a31-4acc1bde12c0 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:26.213862 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 402/810] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Aug 28 00:42:26 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:26.214051 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25f92598-c89e-46de-8a31-4acc1bde12c0 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:26.242991 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:26.262922 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bedad59e-6342-4c40-8b0d-d339cc800fd4 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:26.297474 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb14913a-00f7-4cb3-85ca-29cfe7508730 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:26.298135 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb14913a-00f7-4cb3-85ca-29cfe7508730 None admin] REQUEST_METHOD: `PUT` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:26.298411 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb14913a-00f7-4cb3-85ca-29cfe7508730 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:26.298679 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb14913a-00f7-4cb3-85ca-29cfe7508730 None admin] PATH_INFO: `/v3/projects/f53e6fdb369b480fbebb4086a505df67/users/eede08f020e04282bb994b855e371433/roles/348070fea3dd4cc2a6e00e29fe82cc25` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:26.313438 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb14913a-00f7-4cb3-85ca-29cfe7508730 None admin] RBAC: Authorizing `identity:create_grant(project_id=f53e6fdb369b480fbebb4086a505df67, user_id=eede08f020e04282bb994b855e371433, role_id=348070fea3dd4cc2a6e00e29fe82cc25)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:26.314881 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb14913a-00f7-4cb3-85ca-29cfe7508730 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:26.322576 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25f92598-c89e-46de-8a31-4acc1bde12c0 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:26.330985 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 409/811] 10.222.0.42 () {58 vars in 1439 bytes} [Sun Aug 28 00:42:26 2022] PUT /identity/v3/projects/f53e6fdb369b480fbebb4086a505df67/users/eede08f020e04282bb994b855e371433/roles/348070fea3dd4cc2a6e00e29fe82cc25 => generated 0 bytes in 199 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:42:26.332031 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb14913a-00f7-4cb3-85ca-29cfe7508730 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:26.378209 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb2d559c-bfc6-4182-aae1-43825c65434f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:26.379105 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb2d559c-bfc6-4182-aae1-43825c65434f None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:26.379595 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb2d559c-bfc6-4182-aae1-43825c65434f None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:26.379926 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb2d559c-bfc6-4182-aae1-43825c65434f None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:26.380532 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb2d559c-bfc6-4182-aae1-43825c65434f None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:26.382236 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb2d559c-bfc6-4182-aae1-43825c65434f None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:26.392383 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-cb2d559c-bfc6-4182-aae1-43825c65434f None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 28 00:42:26.392383 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:26.392383 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 28 00:42:26.392383 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 28 00:42:26.392383 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 28 00:42:26.392383 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 28 00:42:26.392383 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 28 00:42:26.392383 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 28 00:42:26.392383 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 28 00:42:26.392383 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:42:26.392383 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 28 00:42:26.392383 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:42:26.392383 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:26.392383 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:26.392383 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:26.392383 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:26.392383 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:26.393490 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:26.393490 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:26.393490 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:26.393490 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:26.393490 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:26.393490 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:26.393490 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 28 00:42:26.393490 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 28 00:42:26.393490 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 28 00:42:26.393490 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 28 00:42:26.393490 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:26.393490 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:26.393490 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:26.393490 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:26.393490 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:26.393490 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:26.395021 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:26.395021 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:26.395021 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:26.395021 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:26.395021 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:26.395021 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:26.395021 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:26.395021 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:26.395021 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:26.395021 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:26.395021 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 28 00:42:26.395021 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 28 00:42:26.395021 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Aug 28 00:42:26.395021 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:42:26.395021 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 403/812] 10.222.0.42 () {56 vars in 1129 bytes} [Sun Aug 28 00:42:26 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:26.395809 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb2d559c-bfc6-4182-aae1-43825c65434f None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:26.455921 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:26.483819 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb14913a-00f7-4cb3-85ca-29cfe7508730 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:26.525167 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9132087e-8374-49fa-a186-f06380512334 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:26.525793 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9132087e-8374-49fa-a186-f06380512334 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:26.526073 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9132087e-8374-49fa-a186-f06380512334 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:26.526610 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9132087e-8374-49fa-a186-f06380512334 None admin] PATH_INFO: `/v3/services` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:26.526977 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9132087e-8374-49fa-a186-f06380512334 None admin] RBAC: Adding query filter params (type=network) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:26.527278 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9132087e-8374-49fa-a186-f06380512334 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:26.528670 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9132087e-8374-49fa-a186-f06380512334 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:26.533723 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:26.536451 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 410/813] 10.222.0.42 () {56 vars in 1112 bytes} [Sun Aug 28 00:42:26 2022] GET /identity/v3/services?type=network => generated 350 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:26.537341 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9132087e-8374-49fa-a186-f06380512334 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:26.551097 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb2d559c-bfc6-4182-aae1-43825c65434f None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:26.587303 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25e554aa-744c-4813-a086-e80975208759 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:26.587969 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25e554aa-744c-4813-a086-e80975208759 None admin] REQUEST_METHOD: `PUT` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:26.588245 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25e554aa-744c-4813-a086-e80975208759 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:26.588513 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25e554aa-744c-4813-a086-e80975208759 None admin] PATH_INFO: `/v3/projects/f53e6fdb369b480fbebb4086a505df67/users/ac4c5584582f4f55a7085af4cbbabf9b/roles/f08d6043261747e3b4763c744eef7dfb` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:26.590616 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25e554aa-744c-4813-a086-e80975208759 None admin] RBAC: Authorizing `identity:create_grant(project_id=f53e6fdb369b480fbebb4086a505df67, user_id=ac4c5584582f4f55a7085af4cbbabf9b, role_id=f08d6043261747e3b4763c744eef7dfb)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:26.592222 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25e554aa-744c-4813-a086-e80975208759 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:26.607063 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 404/814] 10.222.0.42 () {58 vars in 1439 bytes} [Sun Aug 28 00:42:26 2022] PUT /identity/v3/projects/f53e6fdb369b480fbebb4086a505df67/users/ac4c5584582f4f55a7085af4cbbabf9b/roles/f08d6043261747e3b4763c744eef7dfb => generated 0 bytes in 212 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:42:26.608619 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25e554aa-744c-4813-a086-e80975208759 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:26.684416 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9132087e-8374-49fa-a186-f06380512334 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:26.724690 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25e554aa-744c-4813-a086-e80975208759 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:26.724903 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3721af72-4d9d-4f8a-bc74-6bb1c84c05f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:26.725646 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3721af72-4d9d-4f8a-bc74-6bb1c84c05f6 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:26.726059 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3721af72-4d9d-4f8a-bc74-6bb1c84c05f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:26.726461 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3721af72-4d9d-4f8a-bc74-6bb1c84c05f6 None admin] PATH_INFO: `/v3/services` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:26.727440 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3721af72-4d9d-4f8a-bc74-6bb1c84c05f6 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:26.727897 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3721af72-4d9d-4f8a-bc74-6bb1c84c05f6 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:26.728384 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:26.730833 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3721af72-4d9d-4f8a-bc74-6bb1c84c05f6 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:26.739625 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 411/815] 10.222.0.42 () {56 vars in 1124 bytes} [Sun Aug 28 00:42:26 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:26.742426 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd54aa66-84bd-4cde-8906-989324ba3a98 None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:26.742921 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd54aa66-84bd-4cde-8906-989324ba3a98 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:26.743414 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd54aa66-84bd-4cde-8906-989324ba3a98 None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:26.746561 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 412/816] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:42:26 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:42:26.747428 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd54aa66-84bd-4cde-8906-989324ba3a98 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:26.767628 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fff5de2-54c1-4bd2-adee-6b6569513d59 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:26.768428 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fff5de2-54c1-4bd2-adee-6b6569513d59 None admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:26.768829 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fff5de2-54c1-4bd2-adee-6b6569513d59 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:26.769221 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fff5de2-54c1-4bd2-adee-6b6569513d59 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:26.770099 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fff5de2-54c1-4bd2-adee-6b6569513d59 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:26.771770 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fff5de2-54c1-4bd2-adee-6b6569513d59 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:26.799763 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 405/817] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 00:42:26 2022] POST /identity/v3/endpoints => generated 337 bytes in 192 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:42:26.801509 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2511e2ba-89d5-4ce9-9854-fafb3be0d2ce None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:26.801936 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2511e2ba-89d5-4ce9-9854-fafb3be0d2ce None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:26.802935 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2511e2ba-89d5-4ce9-9854-fafb3be0d2ce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:26.825650 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-2511e2ba-89d5-4ce9-9854-fafb3be0d2ce None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:42:26.827002 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-2511e2ba-89d5-4ce9-9854-fafb3be0d2ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:42:26.828499 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:26.896914 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd54aa66-84bd-4cde-8906-989324ba3a98 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:26.940726 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 406/818] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:42:26 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 28 00:42:26.947073 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4eaccf7-ba7a-48b8-a0b9-5b56130918b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:26.948045 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4eaccf7-ba7a-48b8-a0b9-5b56130918b9 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:26.948563 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4eaccf7-ba7a-48b8-a0b9-5b56130918b9 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:26.949084 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4eaccf7-ba7a-48b8-a0b9-5b56130918b9 None admin] PATH_INFO: `/v3/services` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:26.949963 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4eaccf7-ba7a-48b8-a0b9-5b56130918b9 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:26.950469 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4eaccf7-ba7a-48b8-a0b9-5b56130918b9 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:26.952255 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4eaccf7-ba7a-48b8-a0b9-5b56130918b9 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:26.959075 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 413/819] 10.222.0.42 () {56 vars in 1124 bytes} [Sun Aug 28 00:42:26 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:26.964616 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2511e2ba-89d5-4ce9-9854-fafb3be0d2ce None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:27.074639 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:27.099113 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2511e2ba-89d5-4ce9-9854-fafb3be0d2ce None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:27.136145 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50f221f7-5d46-4829-a410-9b667882f810 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:27.136860 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50f221f7-5d46-4829-a410-9b667882f810 None admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:27.137168 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50f221f7-5d46-4829-a410-9b667882f810 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:27.137438 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50f221f7-5d46-4829-a410-9b667882f810 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:27.138105 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50f221f7-5d46-4829-a410-9b667882f810 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:27.139561 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50f221f7-5d46-4829-a410-9b667882f810 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:27.163291 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 407/820] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 00:42:26 2022] POST /identity/v3/endpoints => generated 346 bytes in 199 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:42:27.288022 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4eaccf7-ba7a-48b8-a0b9-5b56130918b9 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:27.424892 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4eaccf7-ba7a-48b8-a0b9-5b56130918b9 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:27.452750 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:27.471927 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebe6a16b-1127-4fd1-96a4-4def013c00d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:27.472592 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebe6a16b-1127-4fd1-96a4-4def013c00d1 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:27.472871 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebe6a16b-1127-4fd1-96a4-4def013c00d1 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:27.473167 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebe6a16b-1127-4fd1-96a4-4def013c00d1 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:27.484971 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebe6a16b-1127-4fd1-96a4-4def013c00d1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 28 00:42:27.491682 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-ebe6a16b-1127-4fd1-96a4-4def013c00d1 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 28 00:42:27.491682 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:27.491682 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:27.491682 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:27.491682 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:27.491682 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:27.491682 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:27.491682 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:27.491682 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:27.491682 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:27.491682 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:27.491682 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:27.491682 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 28 00:42:27.491682 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 28 00:42:27.491682 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 28 00:42:27.491682 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 28 00:42:27.492661 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 28 00:42:27.492661 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 28 00:42:27.492661 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 28 00:42:27.492661 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 28 00:42:27.492661 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:27.492661 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:27.492661 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:27.492661 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:27.492661 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:27.492661 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:27.492661 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:27.492661 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:27.492661 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:27.492661 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:27.492661 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:27.492661 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:27.493548 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:27.493548 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:27.493548 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:27.493548 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:27.493548 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 28 00:42:27.493548 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 28 00:42:27.493548 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 28 00:42:27.493548 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 28 00:42:27.493548 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 28 00:42:27.493548 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 28 00:42:27.493548 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 28 00:42:27.493548 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:27.494254 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 414/821] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Aug 28 00:42:27 2022] GET /identity/v3/roles/admin => generated 83 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:27.499598 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50f221f7-5d46-4829-a410-9b667882f810 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:27.615106 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50f221f7-5d46-4829-a410-9b667882f810 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:27.649230 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-152c485a-95bf-4d2f-b94a-3f7766cbb56a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:27.649842 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-152c485a-95bf-4d2f-b94a-3f7766cbb56a None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:27.650123 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-152c485a-95bf-4d2f-b94a-3f7766cbb56a None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:27.650395 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-152c485a-95bf-4d2f-b94a-3f7766cbb56a None admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:27.651058 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-152c485a-95bf-4d2f-b94a-3f7766cbb56a None admin] RBAC: Adding query filter params (name=admin) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:27.651357 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-152c485a-95bf-4d2f-b94a-3f7766cbb56a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:27.652734 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-152c485a-95bf-4d2f-b94a-3f7766cbb56a None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:27.661239 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 408/822] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Aug 28 00:42:27 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 28 00:42:27.661525 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:27.670617 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebe6a16b-1127-4fd1-96a4-4def013c00d1 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:27.772332 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebe6a16b-1127-4fd1-96a4-4def013c00d1 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:27.808739 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60bc7e28-2a77-4bf4-b7f4-ff3f3bf2a238 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:27.809403 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60bc7e28-2a77-4bf4-b7f4-ff3f3bf2a238 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:27.809680 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60bc7e28-2a77-4bf4-b7f4-ff3f3bf2a238 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:27.809943 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60bc7e28-2a77-4bf4-b7f4-ff3f3bf2a238 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:27.822358 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60bc7e28-2a77-4bf4-b7f4-ff3f3bf2a238 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:27.823821 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60bc7e28-2a77-4bf4-b7f4-ff3f3bf2a238 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:27.839169 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-60bc7e28-2a77-4bf4-b7f4-ff3f3bf2a238 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:42:27.839169 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:27.839169 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:27.839169 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:27.839169 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:27.839169 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:27.839169 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:27.839169 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:27.839169 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:27.839169 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:27.839169 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:27.839169 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:27.839169 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 28 00:42:27.839169 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 28 00:42:27.839169 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 28 00:42:27.839169 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 28 00:42:27.840373 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:27.840373 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:27.840373 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:27.840373 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:27.840373 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:27.840373 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:27.840373 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:27.840373 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:27.840373 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:27.840373 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:27.840373 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:27.840373 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:27.840373 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:27.840373 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:27.840373 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:27.840373 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:27.840373 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 28 00:42:27.841647 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 28 00:42:27.841647 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:42:27.841647 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:27.841647 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 415/823] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:42:27 2022] GET /identity/v3/domains/Default => generated 87 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:27.847851 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-152c485a-95bf-4d2f-b94a-3f7766cbb56a None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:27.952913 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-152c485a-95bf-4d2f-b94a-3f7766cbb56a None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:27.995261 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbb15882-160d-4262-95cf-67725489a397 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:27.995878 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbb15882-160d-4262-95cf-67725489a397 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:27.996156 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbb15882-160d-4262-95cf-67725489a397 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:27.996429 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbb15882-160d-4262-95cf-67725489a397 None admin] PATH_INFO: `/v3/domains` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:27.997101 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbb15882-160d-4262-95cf-67725489a397 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:27.997402 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbb15882-160d-4262-95cf-67725489a397 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:27.998823 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbb15882-160d-4262-95cf-67725489a397 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:28.011471 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 409/824] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Aug 28 00:42:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:28.017432 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60bc7e28-2a77-4bf4-b7f4-ff3f3bf2a238 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:28.050375 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:28.132303 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60bc7e28-2a77-4bf4-b7f4-ff3f3bf2a238 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:28.167779 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0df20ae6-46e2-407f-bb4f-58e3331de11f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:28.168821 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0df20ae6-46e2-407f-bb4f-58e3331de11f None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:28.169522 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0df20ae6-46e2-407f-bb4f-58e3331de11f None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:28.169996 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0df20ae6-46e2-407f-bb4f-58e3331de11f None admin] PATH_INFO: `/v3/users/nova` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:28.185057 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0df20ae6-46e2-407f-bb4f-58e3331de11f None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:28.185057 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0df20ae6-46e2-407f-bb4f-58e3331de11f None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:28.193317 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-0df20ae6-46e2-407f-bb4f-58e3331de11f None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 28 00:42:28.193317 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:28.193317 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:28.193317 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:28.193317 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:28.193317 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:28.193317 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:28.193317 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:28.193317 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:28.193317 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:28.193317 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:28.193317 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:28.193317 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 28 00:42:28.193317 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 28 00:42:28.193317 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 28 00:42:28.193317 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 28 00:42:28.194318 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:28.194318 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:28.194318 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 28 00:42:28.194318 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 28 00:42:28.194318 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 28 00:42:28.194318 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 28 00:42:28.194318 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:28.194318 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:28.194318 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:28.194318 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:28.194318 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:28.194318 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:28.194318 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:28.194318 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:28.194318 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:28.194318 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:28.194318 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:28.194318 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:28.195693 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:28.195693 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:28.195693 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 28 00:42:28.195693 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 28 00:42:28.195693 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 28 00:42:28.195693 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 28 00:42:28.195693 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 28 00:42:28.195693 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 28 00:42:28.195693 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 28 00:42:28.195693 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:28.195693 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 416/825] 10.222.0.42 () {56 vars in 1093 bytes} [Sun Aug 28 00:42:28 2022] GET /identity/v3/users/nova => generated 82 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:28.206255 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbb15882-160d-4262-95cf-67725489a397 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:28.267876 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:28.317222 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbb15882-160d-4262-95cf-67725489a397 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:28.354483 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21106d17-162a-4eef-87a4-b1cdb301a80a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:28.355471 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21106d17-162a-4eef-87a4-b1cdb301a80a None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:28.355978 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21106d17-162a-4eef-87a4-b1cdb301a80a None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:28.356444 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21106d17-162a-4eef-87a4-b1cdb301a80a None admin] PATH_INFO: `/v3/users` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:28.357411 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21106d17-162a-4eef-87a4-b1cdb301a80a None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:28.357950 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21106d17-162a-4eef-87a4-b1cdb301a80a None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:28.359702 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21106d17-162a-4eef-87a4-b1cdb301a80a None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:28.380267 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 410/826] 10.222.0.42 () {56 vars in 1133 bytes} [Sun Aug 28 00:42:28 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:28.385764 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0df20ae6-46e2-407f-bb4f-58e3331de11f None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:28.419535 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2b3bb6f-bac6-4d5d-985b-30867c00d4a8 None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:28.420163 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2b3bb6f-bac6-4d5d-985b-30867c00d4a8 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:28.420694 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2b3bb6f-bac6-4d5d-985b-30867c00d4a8 None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:28.422855 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 411/827] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:42:28 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 28 00:42:28.429299 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b0b7802-90c7-4202-a75f-f67f36fb1311 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:28.429923 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b0b7802-90c7-4202-a75f-f67f36fb1311 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:28.430537 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b0b7802-90c7-4202-a75f-f67f36fb1311 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:28.450771 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-0b0b7802-90c7-4202-a75f-f67f36fb1311 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:42:28.452494 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-0b0b7802-90c7-4202-a75f-f67f36fb1311 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:42:28.484051 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:28.497610 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0df20ae6-46e2-407f-bb4f-58e3331de11f None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:28.538865 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6113ef22-c2e3-47dc-8a4a-683573870245 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:28.539649 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6113ef22-c2e3-47dc-8a4a-683573870245 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:28.539929 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6113ef22-c2e3-47dc-8a4a-683573870245 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:28.540193 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6113ef22-c2e3-47dc-8a4a-683573870245 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:28.553678 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 412/828] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:42:28 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 28 00:42:28.613424 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6113ef22-c2e3-47dc-8a4a-683573870245 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:28.614994 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6113ef22-c2e3-47dc-8a4a-683573870245 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:28.710916 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:28.722397 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 417/829] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Aug 28 00:42:28 2022] GET /identity/v3/auth/tokens => generated 1161 bytes in 337 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 28 00:42:28.727930 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b0b7802-90c7-4202-a75f-f67f36fb1311 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:28.775753 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:28.812623 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5511c865-cbee-464d-ab52-6d290e5353c9 None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:28.812949 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5511c865-cbee-464d-ab52-6d290e5353c9 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:28.814070 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5511c865-cbee-464d-ab52-6d290e5353c9 None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:28.814897 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 418/830] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:42:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:42:28.822336 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17898d09-14ca-4c33-b4d5-4b2c1a0429db None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:28.822737 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17898d09-14ca-4c33-b4d5-4b2c1a0429db None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:28.823110 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17898d09-14ca-4c33-b4d5-4b2c1a0429db None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:28.837775 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b0b7802-90c7-4202-a75f-f67f36fb1311 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:28.846748 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-17898d09-14ca-4c33-b4d5-4b2c1a0429db None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:42:28.848143 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-17898d09-14ca-4c33-b4d5-4b2c1a0429db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:42:28.878333 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e8feab0-9a92-4985-a701-79046e74e688 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:28.878965 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e8feab0-9a92-4985-a701-79046e74e688 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:28.879237 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e8feab0-9a92-4985-a701-79046e74e688 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:28.879499 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e8feab0-9a92-4985-a701-79046e74e688 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:28.890215 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e8feab0-9a92-4985-a701-79046e74e688 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:28.892118 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e8feab0-9a92-4985-a701-79046e74e688 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:28.907493 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-4e8feab0-9a92-4985-a701-79046e74e688 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:42:28.907493 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:28.907493 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:28.907493 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:28.907493 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:28.907493 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:28.907493 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:28.907493 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:28.907493 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:28.907493 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:28.907493 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:28.907493 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:28.907493 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 28 00:42:28.907493 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 28 00:42:28.907493 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 28 00:42:28.907493 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 28 00:42:28.908706 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:28.908706 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:28.908706 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:28.908706 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:28.908706 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:28.908706 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:28.908706 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:28.908706 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:28.908706 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:28.908706 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:28.908706 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:28.908706 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:28.908706 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:28.908706 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:28.908706 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:28.908706 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:28.908706 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 28 00:42:28.909649 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 28 00:42:28.909649 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:42:28.909649 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:42:28.909649 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 413/831] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:42:28 2022] GET /identity/v3/domains/Default => generated 87 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:28.909649 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e8feab0-9a92-4985-a701-79046e74e688 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:28.933573 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 419/832] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:42:28 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 28 00:42:28.934093 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17898d09-14ca-4c33-b4d5-4b2c1a0429db None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:29.014619 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:29.020806 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e8feab0-9a92-4985-a701-79046e74e688 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:29.046638 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17898d09-14ca-4c33-b4d5-4b2c1a0429db None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:29.056873 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1125773-20e3-4ee3-8b4a-f5254a7143a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:29.057771 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1125773-20e3-4ee3-8b4a-f5254a7143a6 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:29.058199 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1125773-20e3-4ee3-8b4a-f5254a7143a6 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:29.058710 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1125773-20e3-4ee3-8b4a-f5254a7143a6 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:29.068979 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:29.070779 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1125773-20e3-4ee3-8b4a-f5254a7143a6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 28 00:42:29.077366 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-d1125773-20e3-4ee3-8b4a-f5254a7143a6 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 28 00:42:29.077366 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:29.077366 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:29.077366 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:29.077366 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:29.077366 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:29.077366 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:29.077366 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:29.077366 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:29.077366 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:29.077366 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:29.077366 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:29.077366 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 28 00:42:29.077366 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 28 00:42:29.077366 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 28 00:42:29.077366 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 28 00:42:29.078562 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 28 00:42:29.078562 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 28 00:42:29.078562 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 28 00:42:29.078562 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 28 00:42:29.078562 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:29.078562 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:29.078562 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:29.078562 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:29.078562 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:29.078562 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:29.078562 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:29.078562 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:29.078562 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:29.078562 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:29.078562 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:29.078562 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:29.079617 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:29.079617 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:29.079617 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:29.079617 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:29.079617 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 28 00:42:29.079617 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 28 00:42:29.079617 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 28 00:42:29.079617 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 28 00:42:29.079617 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 28 00:42:29.079617 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 28 00:42:29.079617 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 28 00:42:29.079617 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:42:29.086946 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 414/833] 10.222.0.42 () {56 vars in 1102 bytes} [Sun Aug 28 00:42:28 2022] GET /identity/v3/roles/service => generated 85 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:29.088135 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1125773-20e3-4ee3-8b4a-f5254a7143a6 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:29.091188 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2bf0721-30d4-454f-94b0-8ffb22b12e7b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:29.091805 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2bf0721-30d4-454f-94b0-8ffb22b12e7b None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:29.092110 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2bf0721-30d4-454f-94b0-8ffb22b12e7b None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:29.092442 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2bf0721-30d4-454f-94b0-8ffb22b12e7b None admin] PATH_INFO: `/v3/domains` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:29.093118 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2bf0721-30d4-454f-94b0-8ffb22b12e7b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:29.093416 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2bf0721-30d4-454f-94b0-8ffb22b12e7b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:29.094832 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2bf0721-30d4-454f-94b0-8ffb22b12e7b None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:29.107008 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 420/834] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Aug 28 00:42:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:29.116060 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2bf0721-30d4-454f-94b0-8ffb22b12e7b None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:29.205439 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1125773-20e3-4ee3-8b4a-f5254a7143a6 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:29.238704 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2bf0721-30d4-454f-94b0-8ffb22b12e7b None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:29.242655 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03b6d9cd-22df-4e47-a609-f5f345d4d3c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:29.243503 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03b6d9cd-22df-4e47-a609-f5f345d4d3c6 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:29.243959 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03b6d9cd-22df-4e47-a609-f5f345d4d3c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:29.244411 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03b6d9cd-22df-4e47-a609-f5f345d4d3c6 None admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:29.245279 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03b6d9cd-22df-4e47-a609-f5f345d4d3c6 None admin] RBAC: Adding query filter params (name=service) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:29.246157 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03b6d9cd-22df-4e47-a609-f5f345d4d3c6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:29.248058 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03b6d9cd-22df-4e47-a609-f5f345d4d3c6 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:29.261006 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 415/835] 10.222.0.42 () {56 vars in 1103 bytes} [Sun Aug 28 00:42:29 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:29.261006 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03b6d9cd-22df-4e47-a609-f5f345d4d3c6 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:29.277545 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-029ee69e-3664-4d82-92ec-d2fccaab4f07 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:29.278173 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-029ee69e-3664-4d82-92ec-d2fccaab4f07 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:29.278449 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-029ee69e-3664-4d82-92ec-d2fccaab4f07 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:29.278711 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-029ee69e-3664-4d82-92ec-d2fccaab4f07 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:29.280256 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:29.285842 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-029ee69e-3664-4d82-92ec-d2fccaab4f07 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:29.287279 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-029ee69e-3664-4d82-92ec-d2fccaab4f07 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:29.298865 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-029ee69e-3664-4d82-92ec-d2fccaab4f07 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 28 00:42:29.298865 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:29.298865 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:29.298865 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:29.298865 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:29.298865 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:29.298865 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:29.298865 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:29.298865 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:29.298865 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:29.298865 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:29.298865 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:29.298865 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 28 00:42:29.298865 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 28 00:42:29.298865 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 28 00:42:29.299792 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 28 00:42:29.299792 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:29.299792 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:29.299792 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:29.299792 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:29.299792 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:29.299792 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:29.299792 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:29.299792 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:29.299792 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:29.299792 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:29.299792 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:29.299792 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:29.299792 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:29.299792 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:29.299792 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:29.299792 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:29.300692 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 28 00:42:29.300692 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 28 00:42:29.300692 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 28 00:42:29.300692 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 28 00:42:29.300692 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 28 00:42:29.300692 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 28 00:42:29.300692 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 28 00:42:29.300692 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:29.300692 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 421/836] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Aug 28 00:42:29 2022] GET /identity/v3/projects/service => generated 88 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:29.300692 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-029ee69e-3664-4d82-92ec-d2fccaab4f07 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:29.341989 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:29.366353 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03b6d9cd-22df-4e47-a609-f5f345d4d3c6 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:29.400510 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d34dee2a-ef99-4dd5-af87-aa7614967ac6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:29.400930 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-029ee69e-3664-4d82-92ec-d2fccaab4f07 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:29.401455 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d34dee2a-ef99-4dd5-af87-aa7614967ac6 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:29.401455 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d34dee2a-ef99-4dd5-af87-aa7614967ac6 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:29.401695 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d34dee2a-ef99-4dd5-af87-aa7614967ac6 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:29.412183 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d34dee2a-ef99-4dd5-af87-aa7614967ac6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 28 00:42:29.417518 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-d34dee2a-ef99-4dd5-af87-aa7614967ac6 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 28 00:42:29.417518 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:29.417518 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:29.417518 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:29.417518 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:29.417518 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:29.417518 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:29.417518 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:29.417518 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:29.417518 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:29.417518 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:29.417518 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:29.417518 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 28 00:42:29.417518 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 28 00:42:29.417518 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 28 00:42:29.417518 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 28 00:42:29.418579 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 28 00:42:29.418579 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 28 00:42:29.418579 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 28 00:42:29.418579 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 28 00:42:29.418579 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:29.418579 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:29.418579 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:29.418579 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:29.418579 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:29.418579 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:29.418579 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:29.418579 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:29.418579 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:29.418579 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:29.418579 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:29.418579 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:29.419431 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:29.419431 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:29.419431 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:29.419431 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:29.419431 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 28 00:42:29.419431 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 28 00:42:29.419431 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 28 00:42:29.419431 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 28 00:42:29.419431 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 28 00:42:29.419431 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 28 00:42:29.419431 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 28 00:42:29.419431 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:42:29.419431 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 416/837] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Aug 28 00:42:29 2022] GET /identity/v3/roles/admin => generated 83 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:29.419431 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d34dee2a-ef99-4dd5-af87-aa7614967ac6 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:29.435575 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b738b259-67bb-4e20-b818-22b8da060d5f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:29.436248 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b738b259-67bb-4e20-b818-22b8da060d5f None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:29.436544 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b738b259-67bb-4e20-b818-22b8da060d5f None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:29.436817 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b738b259-67bb-4e20-b818-22b8da060d5f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:29.447500 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b738b259-67bb-4e20-b818-22b8da060d5f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:29.449002 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b738b259-67bb-4e20-b818-22b8da060d5f None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:29.463872 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-b738b259-67bb-4e20-b818-22b8da060d5f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:42:29.463872 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:29.463872 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:29.463872 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:29.463872 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:29.463872 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:29.463872 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:29.463872 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:29.463872 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:29.463872 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:29.463872 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:29.463872 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:29.463872 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 28 00:42:29.463872 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 28 00:42:29.463872 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 28 00:42:29.463872 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 28 00:42:29.465274 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:29.465274 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:29.465274 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:29.465274 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:29.465274 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:29.465274 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:29.465274 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:29.465274 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:29.465274 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:29.465274 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:29.465274 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:29.465274 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:29.465274 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:29.465274 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:29.465274 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:29.465274 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:29.465274 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 28 00:42:29.466352 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 28 00:42:29.466352 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:42:29.466352 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:29.466352 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 422/838] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:42:29 2022] GET /identity/v3/domains/Default => generated 87 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:29.466352 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ccf3c02-9080-453d-a426-d661a2ff5e9a None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:29.466634 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ccf3c02-9080-453d-a426-d661a2ff5e9a None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:29.466825 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ccf3c02-9080-453d-a426-d661a2ff5e9a None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:29.468097 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 423/839] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:42:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:42:29.468821 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ccf3c02-9080-453d-a426-d661a2ff5e9a None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:29.506811 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:29.527111 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d34dee2a-ef99-4dd5-af87-aa7614967ac6 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:29.560385 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f1b1cdc-e861-4f83-a580-71599a5e40a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:29.560994 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f1b1cdc-e861-4f83-a580-71599a5e40a8 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:29.561277 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f1b1cdc-e861-4f83-a580-71599a5e40a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:29.561552 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f1b1cdc-e861-4f83-a580-71599a5e40a8 None admin] PATH_INFO: `/v3/projects` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:29.562245 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f1b1cdc-e861-4f83-a580-71599a5e40a8 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:29.562554 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f1b1cdc-e861-4f83-a580-71599a5e40a8 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:29.564023 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f1b1cdc-e861-4f83-a580-71599a5e40a8 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:29.564515 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:29.575265 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ccf3c02-9080-453d-a426-d661a2ff5e9a None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:29.576312 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 417/840] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Aug 28 00:42:29 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:29.577095 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f1b1cdc-e861-4f83-a580-71599a5e40a8 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:29.609757 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7eae3d4-bf20-4746-96cd-85f18dd42e1d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:29.610379 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7eae3d4-bf20-4746-96cd-85f18dd42e1d None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:29.610656 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7eae3d4-bf20-4746-96cd-85f18dd42e1d None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:29.610925 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7eae3d4-bf20-4746-96cd-85f18dd42e1d None admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:29.611580 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7eae3d4-bf20-4746-96cd-85f18dd42e1d None admin] RBAC: Adding query filter params (name=admin) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:29.611883 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7eae3d4-bf20-4746-96cd-85f18dd42e1d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:29.613295 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7eae3d4-bf20-4746-96cd-85f18dd42e1d None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:29.621512 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 424/841] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Aug 28 00:42:29 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:29.622706 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b275caa-2758-4b08-badf-ce2ccd431b13 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:29.622992 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b275caa-2758-4b08-badf-ce2ccd431b13 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:29.623265 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b275caa-2758-4b08-badf-ce2ccd431b13 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:29.640992 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-5b275caa-2758-4b08-badf-ce2ccd431b13 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:42:29.642217 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-5b275caa-2758-4b08-badf-ce2ccd431b13 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:42:29.681509 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f1b1cdc-e861-4f83-a580-71599a5e40a8 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:29.714603 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 425/842] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:42:29 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 28 00:42:29.715469 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b275caa-2758-4b08-badf-ce2ccd431b13 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:29.723669 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9116085a-bd94-4053-88a2-33ab86898995 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:29.724285 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9116085a-bd94-4053-88a2-33ab86898995 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:29.724596 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9116085a-bd94-4053-88a2-33ab86898995 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:29.724871 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9116085a-bd94-4053-88a2-33ab86898995 None admin] PATH_INFO: `/v3/domains` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:29.725577 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9116085a-bd94-4053-88a2-33ab86898995 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:29.725884 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9116085a-bd94-4053-88a2-33ab86898995 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:29.727332 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9116085a-bd94-4053-88a2-33ab86898995 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:29.736297 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:29.744106 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 418/843] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Aug 28 00:42:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:29.744106 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9116085a-bd94-4053-88a2-33ab86898995 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:29.814644 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:29.832098 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b275caa-2758-4b08-badf-ce2ccd431b13 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:29.849667 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9116085a-bd94-4053-88a2-33ab86898995 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:29.869635 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6a885ff-d125-4a6b-a3c7-a90b1adffbcf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:29.870251 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6a885ff-d125-4a6b-a3c7-a90b1adffbcf None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:29.870537 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6a885ff-d125-4a6b-a3c7-a90b1adffbcf None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:29.870833 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6a885ff-d125-4a6b-a3c7-a90b1adffbcf None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:29.871640 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6a885ff-d125-4a6b-a3c7-a90b1adffbcf None admin] RBAC: Adding query filter params (role.id=f08d6043261747e3b4763c744eef7dfb, scope.project.id=f53e6fdb369b480fbebb4086a505df67, user.id=168b07df50c24593ba73dbde5b037aed) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:29.871946 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6a885ff-d125-4a6b-a3c7-a90b1adffbcf None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:29.873494 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6a885ff-d125-4a6b-a3c7-a90b1adffbcf None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:29.880155 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 426/844] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Aug 28 00:42:29 2022] GET /identity/v3/role_assignments?user.id=168b07df50c24593ba73dbde5b037aed&scope.project.id=f53e6fdb369b480fbebb4086a505df67&role.id=f08d6043261747e3b4763c744eef7dfb => generated 608 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:29.881373 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6a885ff-d125-4a6b-a3c7-a90b1adffbcf None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:29.896628 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d77f41e1-5361-4110-8601-607e37663ecb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:29.897333 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d77f41e1-5361-4110-8601-607e37663ecb None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:29.897596 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d77f41e1-5361-4110-8601-607e37663ecb None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:29.897866 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d77f41e1-5361-4110-8601-607e37663ecb None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:29.918358 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d77f41e1-5361-4110-8601-607e37663ecb None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:29.920470 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d77f41e1-5361-4110-8601-607e37663ecb None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:29.941479 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-d77f41e1-5361-4110-8601-607e37663ecb None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:42:29.941479 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:29.941479 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:29.941479 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:29.941479 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:29.941479 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:29.941479 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:29.941479 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:29.941479 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:29.941479 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:29.941479 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:29.941479 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:29.941479 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 28 00:42:29.941479 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 28 00:42:29.941479 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 28 00:42:29.941479 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 28 00:42:29.942514 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:29.942514 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:29.942514 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:29.942514 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:29.942514 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:29.942514 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:29.942514 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:29.942514 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:29.942514 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:29.942514 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:29.942514 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:29.942514 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:29.942514 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:29.942514 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:29.942514 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:29.942514 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:29.942514 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 28 00:42:29.943384 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 28 00:42:29.943384 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:42:29.943384 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:42:29.945126 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 419/845] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:42:29 2022] GET /identity/v3/domains/Default => generated 87 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:29.948674 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d77f41e1-5361-4110-8601-607e37663ecb None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:30.014815 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:30.020336 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6a885ff-d125-4a6b-a3c7-a90b1adffbcf None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:30.060467 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-848c7b58-a984-496b-9829-38ac8e0a9be0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:30.061175 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-848c7b58-a984-496b-9829-38ac8e0a9be0 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:30.061462 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-848c7b58-a984-496b-9829-38ac8e0a9be0 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:30.061730 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-848c7b58-a984-496b-9829-38ac8e0a9be0 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:30.070079 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-848c7b58-a984-496b-9829-38ac8e0a9be0 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:30.071506 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-848c7b58-a984-496b-9829-38ac8e0a9be0 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:30.073678 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d77f41e1-5361-4110-8601-607e37663ecb None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:30.083543 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-848c7b58-a984-496b-9829-38ac8e0a9be0 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 28 00:42:30.083543 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:30.083543 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:30.083543 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:30.083543 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:30.083543 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:30.083543 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:30.083543 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:30.083543 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:30.083543 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:30.083543 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:30.083543 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:30.083543 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 28 00:42:30.083543 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 28 00:42:30.083543 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 28 00:42:30.083543 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 28 00:42:30.084526 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:30.084526 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:30.084526 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 28 00:42:30.084526 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 28 00:42:30.084526 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 28 00:42:30.084526 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 28 00:42:30.084526 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:30.084526 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:30.084526 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:30.084526 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:30.084526 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:30.084526 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:30.084526 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:30.084526 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:30.084526 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:30.084526 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:30.084526 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:30.084526 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:30.085536 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:30.085536 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:30.085536 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 28 00:42:30.085536 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 28 00:42:30.085536 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 28 00:42:30.085536 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 28 00:42:30.085536 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 28 00:42:30.085536 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 28 00:42:30.085536 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 28 00:42:30.085536 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:30.085536 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 427/846] 10.222.0.42 () {56 vars in 1117 bytes} [Sun Aug 28 00:42:29 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:30.085536 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-848c7b58-a984-496b-9829-38ac8e0a9be0 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:30.097692 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:30.123988 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29138fb6-8089-4788-9eab-f7413af8108c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:30.124596 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29138fb6-8089-4788-9eab-f7413af8108c None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:30.124869 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29138fb6-8089-4788-9eab-f7413af8108c None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:30.125159 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29138fb6-8089-4788-9eab-f7413af8108c None admin] PATH_INFO: `/v3/domains` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:30.125852 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29138fb6-8089-4788-9eab-f7413af8108c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:30.126149 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29138fb6-8089-4788-9eab-f7413af8108c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:30.127523 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29138fb6-8089-4788-9eab-f7413af8108c None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:30.141213 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 420/847] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Aug 28 00:42:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 28 00:42:30.141863 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29138fb6-8089-4788-9eab-f7413af8108c None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:30.195441 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-848c7b58-a984-496b-9829-38ac8e0a9be0 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:30.231913 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e4c5cb0-599b-401c-bf97-3d03f8022c44 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:30.232553 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e4c5cb0-599b-401c-bf97-3d03f8022c44 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:30.232890 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e4c5cb0-599b-401c-bf97-3d03f8022c44 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:30.233254 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e4c5cb0-599b-401c-bf97-3d03f8022c44 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:30.233785 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e4c5cb0-599b-401c-bf97-3d03f8022c44 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:30.235158 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e4c5cb0-599b-401c-bf97-3d03f8022c44 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:30.246522 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-2e4c5cb0-599b-401c-bf97-3d03f8022c44 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 28 00:42:30.246522 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:30.246522 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 28 00:42:30.246522 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 28 00:42:30.246522 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 28 00:42:30.246522 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 28 00:42:30.246522 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 28 00:42:30.246522 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 28 00:42:30.246522 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 28 00:42:30.246522 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:30.246522 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 28 00:42:30.246522 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:30.246522 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:30.246522 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:30.246522 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:30.246522 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:30.246522 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:30.247599 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:30.247599 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:30.247599 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:30.247599 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:30.247599 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:30.247599 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:30.247599 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 28 00:42:30.247599 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 28 00:42:30.247599 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 28 00:42:30.247599 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 28 00:42:30.247599 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:30.247599 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:30.247599 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:30.247599 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:30.247599 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:30.247599 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:30.248505 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:30.248505 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:30.248505 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:30.248505 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:30.248505 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:30.248505 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:30.248505 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:30.248505 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:30.248505 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:30.248505 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:30.248505 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 28 00:42:30.248505 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 28 00:42:30.248505 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 28 00:42:30.248505 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:30.249275 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29138fb6-8089-4788-9eab-f7413af8108c None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:30.249350 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 428/848] 10.222.0.42 () {56 vars in 1114 bytes} [Sun Aug 28 00:42:30 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:30.249350 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e4c5cb0-599b-401c-bf97-3d03f8022c44 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:30.266930 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:30.289763 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbb19515-5cb6-4c82-ab24-0575bc1dd835 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:30.290371 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb19515-5cb6-4c82-ab24-0575bc1dd835 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:30.290643 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb19515-5cb6-4c82-ab24-0575bc1dd835 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:30.290908 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb19515-5cb6-4c82-ab24-0575bc1dd835 None admin] PATH_INFO: `/v3/users` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:30.291651 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbb19515-5cb6-4c82-ab24-0575bc1dd835 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:30.291932 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbb19515-5cb6-4c82-ab24-0575bc1dd835 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:30.293399 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbb19515-5cb6-4c82-ab24-0575bc1dd835 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:30.312409 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 421/849] 10.222.0.42 () {56 vars in 1149 bytes} [Sun Aug 28 00:42:30 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:30.313281 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbb19515-5cb6-4c82-ab24-0575bc1dd835 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:30.331071 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:30.362219 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e4c5cb0-599b-401c-bf97-3d03f8022c44 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:30.400710 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c3d7cc2-f034-4373-88fa-e0bae4c9580e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:30.401386 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c3d7cc2-f034-4373-88fa-e0bae4c9580e None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:30.401917 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c3d7cc2-f034-4373-88fa-e0bae4c9580e None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:30.401917 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c3d7cc2-f034-4373-88fa-e0bae4c9580e None admin] PATH_INFO: `/v3/users/swift` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:30.409590 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c3d7cc2-f034-4373-88fa-e0bae4c9580e None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:30.411031 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c3d7cc2-f034-4373-88fa-e0bae4c9580e None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:30.420999 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbb19515-5cb6-4c82-ab24-0575bc1dd835 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:30.422668 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-7c3d7cc2-f034-4373-88fa-e0bae4c9580e None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Aug 28 00:42:30.422668 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:30.422668 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:30.422668 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:30.422668 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:30.422668 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:30.422668 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:30.422668 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:30.422668 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:30.422668 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:30.422668 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:30.422668 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:30.422668 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 28 00:42:30.422668 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 28 00:42:30.422668 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 28 00:42:30.422668 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 28 00:42:30.423614 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:30.423614 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:30.423614 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 28 00:42:30.423614 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 28 00:42:30.423614 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 28 00:42:30.423614 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 28 00:42:30.423614 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:30.423614 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:30.423614 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:30.423614 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:30.423614 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:30.423614 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:30.423614 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:30.423614 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:30.423614 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:30.423614 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:30.423614 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:30.423614 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:30.424655 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:30.424655 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:30.424655 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 28 00:42:30.424655 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 28 00:42:30.424655 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 28 00:42:30.424655 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 28 00:42:30.424655 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 28 00:42:30.424655 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 28 00:42:30.424655 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Aug 28 00:42:30.424655 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:30.424655 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 429/850] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Aug 28 00:42:30 2022] GET /identity/v3/users/swift => generated 83 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:30.424655 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c3d7cc2-f034-4373-88fa-e0bae4c9580e None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:30.466608 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f9e316f-2ff8-4224-b7f1-dd422d0d019a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:30.467254 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f9e316f-2ff8-4224-b7f1-dd422d0d019a None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:30.467529 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f9e316f-2ff8-4224-b7f1-dd422d0d019a None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:30.467792 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f9e316f-2ff8-4224-b7f1-dd422d0d019a None admin] PATH_INFO: `/v3/services` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:30.468413 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f9e316f-2ff8-4224-b7f1-dd422d0d019a None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:30.468710 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f9e316f-2ff8-4224-b7f1-dd422d0d019a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:30.470127 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f9e316f-2ff8-4224-b7f1-dd422d0d019a None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:30.479237 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 422/851] 10.222.0.42 () {56 vars in 1114 bytes} [Sun Aug 28 00:42:30 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:30.479237 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f9e316f-2ff8-4224-b7f1-dd422d0d019a None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:30.517361 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:30.539123 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c3d7cc2-f034-4373-88fa-e0bae4c9580e None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:30.570973 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:30.576721 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ec10e17-8813-4936-9f65-78057f1bb5f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:30.577368 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ec10e17-8813-4936-9f65-78057f1bb5f5 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:30.577650 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ec10e17-8813-4936-9f65-78057f1bb5f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:30.577931 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ec10e17-8813-4936-9f65-78057f1bb5f5 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:30.588273 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f9e316f-2ff8-4224-b7f1-dd422d0d019a None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:30.628194 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e9043ea-9a5f-4cf0-ae45-b50456627bc3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:30.628785 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e9043ea-9a5f-4cf0-ae45-b50456627bc3 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:30.629126 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e9043ea-9a5f-4cf0-ae45-b50456627bc3 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:30.629359 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e9043ea-9a5f-4cf0-ae45-b50456627bc3 None admin] PATH_INFO: `/v3/users` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:30.630143 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e9043ea-9a5f-4cf0-ae45-b50456627bc3 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:30.630435 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e9043ea-9a5f-4cf0-ae45-b50456627bc3 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:30.631860 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e9043ea-9a5f-4cf0-ae45-b50456627bc3 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:30.654056 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 423/852] 10.222.0.42 () {56 vars in 1135 bytes} [Sun Aug 28 00:42:30 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:30.654056 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e9043ea-9a5f-4cf0-ae45-b50456627bc3 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:30.655864 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ec10e17-8813-4936-9f65-78057f1bb5f5 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:30.657397 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ec10e17-8813-4936-9f65-78057f1bb5f5 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:30.744064 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:30.753653 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e9043ea-9a5f-4cf0-ae45-b50456627bc3 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:30.756800 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 430/853] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Aug 28 00:42:30 2022] GET /identity/v3/auth/tokens => generated 1161 bytes in 332 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 28 00:42:30.761888 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ec10e17-8813-4936-9f65-78057f1bb5f5 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:30.789452 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53e2bf72-dfa4-4249-9d95-e8c72d8f5ab6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:30.790429 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53e2bf72-dfa4-4249-9d95-e8c72d8f5ab6 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:30.790954 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53e2bf72-dfa4-4249-9d95-e8c72d8f5ab6 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:30.791324 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53e2bf72-dfa4-4249-9d95-e8c72d8f5ab6 None admin] PATH_INFO: `/v3/services` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:30.792062 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53e2bf72-dfa4-4249-9d95-e8c72d8f5ab6 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:30.792501 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53e2bf72-dfa4-4249-9d95-e8c72d8f5ab6 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:30.793874 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:30.793993 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53e2bf72-dfa4-4249-9d95-e8c72d8f5ab6 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:30.805000 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 424/854] 10.222.0.42 () {56 vars in 1114 bytes} [Sun Aug 28 00:42:30 2022] GET /identity/v3/services?type=volumev3 => generated 125 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:30.805000 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53e2bf72-dfa4-4249-9d95-e8c72d8f5ab6 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:30.904188 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ec10e17-8813-4936-9f65-78057f1bb5f5 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:30.931371 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53e2bf72-dfa4-4249-9d95-e8c72d8f5ab6 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:30.958169 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d67f3450-35c1-467b-8ffd-21c7e5c9051b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:30.959355 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d67f3450-35c1-467b-8ffd-21c7e5c9051b None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:30.959981 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d67f3450-35c1-467b-8ffd-21c7e5c9051b None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:30.959981 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d67f3450-35c1-467b-8ffd-21c7e5c9051b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:30.981114 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1b64eeb-d06b-47b6-a9e8-e6077bd6d4ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:30.981764 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1b64eeb-d06b-47b6-a9e8-e6077bd6d4ef None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:30.982071 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1b64eeb-d06b-47b6-a9e8-e6077bd6d4ef None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:30.982734 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1b64eeb-d06b-47b6-a9e8-e6077bd6d4ef None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:30.997145 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1b64eeb-d06b-47b6-a9e8-e6077bd6d4ef None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:30.998792 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1b64eeb-d06b-47b6-a9e8-e6077bd6d4ef None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:31.014768 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-b1b64eeb-d06b-47b6-a9e8-e6077bd6d4ef None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:42:31.014768 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:31.014768 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:31.014768 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:31.014768 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:31.014768 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:31.014768 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:31.014768 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:31.014768 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:31.014768 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:31.014768 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:31.014768 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:31.014768 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 28 00:42:31.014768 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 28 00:42:31.014768 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 28 00:42:31.014768 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 28 00:42:31.016202 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:31.016202 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:31.016202 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:31.016202 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:31.016202 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:31.016202 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:31.016202 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:31.016202 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:31.016202 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:31.016202 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:31.016202 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:31.016202 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:31.016202 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:31.016202 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:31.016202 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:31.016202 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:31.016202 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 28 00:42:31.017157 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 28 00:42:31.017157 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:42:31.017157 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:42:31.017157 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 425/855] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:42:30 2022] GET /identity/v3/domains/Default => generated 87 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:31.021983 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1b64eeb-d06b-47b6-a9e8-e6077bd6d4ef None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:31.035377 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:31.053556 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d67f3450-35c1-467b-8ffd-21c7e5c9051b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:31.055134 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d67f3450-35c1-467b-8ffd-21c7e5c9051b None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:31.086114 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:31.137444 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1b64eeb-d06b-47b6-a9e8-e6077bd6d4ef None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:31.173696 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 431/856] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Aug 28 00:42:30 2022] GET /identity/v3/auth/tokens => generated 1161 bytes in 412 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 28 00:42:31.174555 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e625618f-a4bb-4c83-adae-b5004e74bbaa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:31.175030 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d67f3450-35c1-467b-8ffd-21c7e5c9051b None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:31.175532 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e625618f-a4bb-4c83-adae-b5004e74bbaa None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:31.175951 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e625618f-a4bb-4c83-adae-b5004e74bbaa None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:31.176363 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e625618f-a4bb-4c83-adae-b5004e74bbaa None admin] PATH_INFO: `/v3/domains` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:31.177189 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e625618f-a4bb-4c83-adae-b5004e74bbaa None admin] RBAC: Adding query filter params (name=Default) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:31.177669 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e625618f-a4bb-4c83-adae-b5004e74bbaa None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:31.179276 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e625618f-a4bb-4c83-adae-b5004e74bbaa None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:31.192329 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 426/857] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Aug 28 00:42:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 28 00:42:31.197620 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e625618f-a4bb-4c83-adae-b5004e74bbaa None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:31.269135 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:31.280779 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d67f3450-35c1-467b-8ffd-21c7e5c9051b None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:31.307342 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e625618f-a4bb-4c83-adae-b5004e74bbaa None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:31.317617 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fd1cad3-a430-4641-884c-76be97a60e21 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:31.318294 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fd1cad3-a430-4641-884c-76be97a60e21 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:31.318855 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fd1cad3-a430-4641-884c-76be97a60e21 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:31.318966 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fd1cad3-a430-4641-884c-76be97a60e21 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:31.329320 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:31.331178 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fd1cad3-a430-4641-884c-76be97a60e21 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:31.332491 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fd1cad3-a430-4641-884c-76be97a60e21 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:31.345883 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e371257b-b76d-428f-b087-6bf9c56792a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:31.346811 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e371257b-b76d-428f-b087-6bf9c56792a1 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:31.347235 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e371257b-b76d-428f-b087-6bf9c56792a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:31.347650 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e371257b-b76d-428f-b087-6bf9c56792a1 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:31.350266 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-0fd1cad3-a430-4641-884c-76be97a60e21 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:42:31.350266 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:31.350266 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:31.350266 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:31.350266 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:31.350266 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:31.350266 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:31.350266 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:31.350266 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:31.350266 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:31.350266 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:31.350266 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:31.350266 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 28 00:42:31.350266 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 28 00:42:31.350266 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 28 00:42:31.350266 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 28 00:42:31.351427 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:31.351427 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:31.351427 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:31.351427 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:31.351427 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:31.351427 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:31.351427 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:31.351427 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:31.351427 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:31.351427 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:31.351427 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:31.351427 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:31.351427 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:31.351427 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:31.351427 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:31.351427 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:31.351427 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 28 00:42:31.352550 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 28 00:42:31.352550 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:42:31.352550 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:31.357873 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e371257b-b76d-428f-b087-6bf9c56792a1 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:31.358106 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 432/858] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:42:31 2022] GET /identity/v3/domains/Default => generated 87 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 28 00:42:31.359447 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fd1cad3-a430-4641-884c-76be97a60e21 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:31.359962 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e371257b-b76d-428f-b087-6bf9c56792a1 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:31.372339 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-e371257b-b76d-428f-b087-6bf9c56792a1 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 28 00:42:31.372339 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:31.372339 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:31.372339 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:31.372339 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:31.372339 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:31.372339 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:31.372339 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:31.372339 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:31.372339 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:31.372339 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:31.372339 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:31.372339 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 28 00:42:31.372339 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 28 00:42:31.372339 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 28 00:42:31.373404 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 28 00:42:31.373404 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:31.373404 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:31.373404 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:31.373404 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:31.373404 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:31.373404 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:31.373404 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:31.373404 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:31.373404 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:31.373404 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:31.373404 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:31.373404 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:31.373404 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:31.373404 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:31.373404 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:31.373404 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:31.374311 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 28 00:42:31.374311 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 28 00:42:31.374311 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 28 00:42:31.374311 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 28 00:42:31.374311 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 28 00:42:31.374311 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 28 00:42:31.374311 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 28 00:42:31.374311 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:42:31.376440 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 427/859] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Aug 28 00:42:31 2022] GET /identity/v3/projects/service => generated 88 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:31.382294 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e371257b-b76d-428f-b087-6bf9c56792a1 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:31.480565 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fd1cad3-a430-4641-884c-76be97a60e21 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:31.526845 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87e26861-7b72-408d-871f-c615f103cd26 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:31.527463 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87e26861-7b72-408d-871f-c615f103cd26 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:31.527732 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87e26861-7b72-408d-871f-c615f103cd26 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:31.528004 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87e26861-7b72-408d-871f-c615f103cd26 None admin] PATH_INFO: `/v3/domains` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:31.528699 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87e26861-7b72-408d-871f-c615f103cd26 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:31.529074 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87e26861-7b72-408d-871f-c615f103cd26 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:31.530587 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87e26861-7b72-408d-871f-c615f103cd26 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:31.533359 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e371257b-b76d-428f-b087-6bf9c56792a1 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:31.548464 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 433/860] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Aug 28 00:42:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:31.549735 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87e26861-7b72-408d-871f-c615f103cd26 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:31.562010 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:31.570723 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c57a9958-41a2-487c-bd1b-65ec8607354b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:31.571704 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c57a9958-41a2-487c-bd1b-65ec8607354b None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:31.572129 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c57a9958-41a2-487c-bd1b-65ec8607354b None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:31.572588 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c57a9958-41a2-487c-bd1b-65ec8607354b None admin] PATH_INFO: `/v3/projects` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:31.573468 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c57a9958-41a2-487c-bd1b-65ec8607354b None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:31.573916 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c57a9958-41a2-487c-bd1b-65ec8607354b None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:31.575707 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c57a9958-41a2-487c-bd1b-65ec8607354b None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:31.590079 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 428/861] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Aug 28 00:42:31 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 28 00:42:31.596190 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c57a9958-41a2-487c-bd1b-65ec8607354b None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:31.607310 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:31.676260 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87e26861-7b72-408d-871f-c615f103cd26 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:31.719386 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f32b83f4-8a3d-4815-a55f-2455540f2a5d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:31.719950 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c57a9958-41a2-487c-bd1b-65ec8607354b None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:31.720451 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f32b83f4-8a3d-4815-a55f-2455540f2a5d None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:31.720872 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f32b83f4-8a3d-4815-a55f-2455540f2a5d None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:31.721307 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f32b83f4-8a3d-4815-a55f-2455540f2a5d None admin] PATH_INFO: `/v3/projects/service` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:31.729807 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f32b83f4-8a3d-4815-a55f-2455540f2a5d None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:31.732143 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f32b83f4-8a3d-4815-a55f-2455540f2a5d None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:31.747476 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-f32b83f4-8a3d-4815-a55f-2455540f2a5d None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 28 00:42:31.747476 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:31.747476 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:31.747476 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:31.747476 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:31.747476 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:31.747476 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:31.747476 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:31.747476 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:31.747476 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:31.747476 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:31.747476 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:31.747476 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 28 00:42:31.747476 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 28 00:42:31.747476 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 28 00:42:31.748629 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 28 00:42:31.748629 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:31.748629 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:31.748629 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:31.748629 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:31.748629 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:31.748629 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:31.748629 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:31.748629 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:31.748629 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:31.748629 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:31.748629 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:31.748629 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:31.748629 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:31.748629 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:31.748629 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:31.748629 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:31.749795 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 28 00:42:31.749795 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 28 00:42:31.749795 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 28 00:42:31.749795 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 28 00:42:31.749795 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 28 00:42:31.749795 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 28 00:42:31.749795 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 28 00:42:31.749795 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:31.752426 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 434/862] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Aug 28 00:42:31 2022] GET /identity/v3/projects/service => generated 88 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:31.759107 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f32b83f4-8a3d-4815-a55f-2455540f2a5d None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:31.767680 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1669c59f-8e71-4f3b-a628-09bbe97b23e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:31.768603 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1669c59f-8e71-4f3b-a628-09bbe97b23e5 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:31.769129 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1669c59f-8e71-4f3b-a628-09bbe97b23e5 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:31.769634 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1669c59f-8e71-4f3b-a628-09bbe97b23e5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:31.771562 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1669c59f-8e71-4f3b-a628-09bbe97b23e5 None admin] RBAC: Adding query filter params (role.id=348070fea3dd4cc2a6e00e29fe82cc25, scope.project.id=f53e6fdb369b480fbebb4086a505df67, user.id=eede08f020e04282bb994b855e371433) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:31.772116 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1669c59f-8e71-4f3b-a628-09bbe97b23e5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:31.775051 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1669c59f-8e71-4f3b-a628-09bbe97b23e5 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:31.789118 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 429/863] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Aug 28 00:42:31 2022] GET /identity/v3/role_assignments?user.id=eede08f020e04282bb994b855e371433&scope.project.id=f53e6fdb369b480fbebb4086a505df67&role.id=348070fea3dd4cc2a6e00e29fe82cc25 => generated 608 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:31.873958 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad89e412-2aaa-45ef-a9e4-7396612cc75e None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:31.874258 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad89e412-2aaa-45ef-a9e4-7396612cc75e None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:31.874510 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad89e412-2aaa-45ef-a9e4-7396612cc75e None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:31.876169 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 430/864] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:42:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 28 00:42:31.882043 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-049b9fb4-6f28-425c-ad05-e9da3a90bfd2 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:31.882313 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-049b9fb4-6f28-425c-ad05-e9da3a90bfd2 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:31.882588 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-049b9fb4-6f28-425c-ad05-e9da3a90bfd2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:31.888692 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f32b83f4-8a3d-4815-a55f-2455540f2a5d None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:31.897386 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:31.902475 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-049b9fb4-6f28-425c-ad05-e9da3a90bfd2 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:42:31.903878 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-049b9fb4-6f28-425c-ad05-e9da3a90bfd2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:42:31.940687 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb42af38-c69d-4c2d-83a9-9817a87ba2f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:31.941898 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb42af38-c69d-4c2d-83a9-9817a87ba2f0 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:31.942397 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb42af38-c69d-4c2d-83a9-9817a87ba2f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:31.943307 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb42af38-c69d-4c2d-83a9-9817a87ba2f0 None admin] PATH_INFO: `/v3/projects` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:31.944593 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb42af38-c69d-4c2d-83a9-9817a87ba2f0 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:31.945113 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb42af38-c69d-4c2d-83a9-9817a87ba2f0 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:31.946769 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb42af38-c69d-4c2d-83a9-9817a87ba2f0 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:31.963461 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 435/865] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Aug 28 00:42:31 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:31.965331 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:31.970261 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb42af38-c69d-4c2d-83a9-9817a87ba2f0 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:31.988353 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 431/866] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:42:31 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 28 00:42:32.084587 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb42af38-c69d-4c2d-83a9-9817a87ba2f0 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:32.126240 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72518c8f-25be-4147-b9bb-8c24899dc367 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:32.126863 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72518c8f-25be-4147-b9bb-8c24899dc367 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:32.127143 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72518c8f-25be-4147-b9bb-8c24899dc367 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:32.127414 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72518c8f-25be-4147-b9bb-8c24899dc367 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:32.128150 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72518c8f-25be-4147-b9bb-8c24899dc367 None admin] RBAC: Adding query filter params (role.id=f08d6043261747e3b4763c744eef7dfb, scope.project.id=f53e6fdb369b480fbebb4086a505df67, user.id=ac4c5584582f4f55a7085af4cbbabf9b) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:32.128456 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72518c8f-25be-4147-b9bb-8c24899dc367 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:32.129923 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72518c8f-25be-4147-b9bb-8c24899dc367 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:32.137975 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 436/867] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Aug 28 00:42:31 2022] GET /identity/v3/role_assignments?user.id=ac4c5584582f4f55a7085af4cbbabf9b&scope.project.id=f53e6fdb369b480fbebb4086a505df67&role.id=f08d6043261747e3b4763c744eef7dfb => generated 608 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 28 00:42:32.256643 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-049b9fb4-6f28-425c-ad05-e9da3a90bfd2 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:32.363995 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-049b9fb4-6f28-425c-ad05-e9da3a90bfd2 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:32.399139 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b2b10d5-55e0-43ac-b37d-2bb99c62bdd5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:32.399777 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b2b10d5-55e0-43ac-b37d-2bb99c62bdd5 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:32.400053 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b2b10d5-55e0-43ac-b37d-2bb99c62bdd5 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:32.400325 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b2b10d5-55e0-43ac-b37d-2bb99c62bdd5 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:32.400909 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b2b10d5-55e0-43ac-b37d-2bb99c62bdd5 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:32.402314 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b2b10d5-55e0-43ac-b37d-2bb99c62bdd5 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:32.411662 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-7b2b10d5-55e0-43ac-b37d-2bb99c62bdd5 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 28 00:42:32.411662 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:32.411662 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 28 00:42:32.411662 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 28 00:42:32.411662 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 28 00:42:32.411662 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 28 00:42:32.411662 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 28 00:42:32.411662 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 28 00:42:32.411662 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 28 00:42:32.411662 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:42:32.411662 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 28 00:42:32.411662 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:42:32.411662 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:32.411662 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:32.411662 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:32.411662 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:32.412661 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:32.412661 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:32.412661 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:32.412661 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:32.412661 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:32.412661 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:32.412661 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:32.412661 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 28 00:42:32.412661 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 28 00:42:32.412661 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 28 00:42:32.412661 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 28 00:42:32.412661 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:32.412661 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:32.412661 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:32.412661 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:32.412661 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:32.412661 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:32.413592 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:32.413592 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:32.413592 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:32.413592 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:32.413592 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:32.413592 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:32.413592 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:32.413592 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:32.413592 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:32.413592 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:32.413592 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 28 00:42:32.413592 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 28 00:42:32.413592 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 28 00:42:32.413592 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:42:32.413592 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 432/868] 10.222.0.42 () {56 vars in 1132 bytes} [Sun Aug 28 00:42:32 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:32.419096 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72518c8f-25be-4147-b9bb-8c24899dc367 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:32.452314 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:32.529595 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72518c8f-25be-4147-b9bb-8c24899dc367 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:32.564820 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-387fc687-96b6-4e08-a7b1-1e9dc5518af5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:32.565462 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-387fc687-96b6-4e08-a7b1-1e9dc5518af5 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:32.565738 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-387fc687-96b6-4e08-a7b1-1e9dc5518af5 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:32.566006 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-387fc687-96b6-4e08-a7b1-1e9dc5518af5 None admin] PATH_INFO: `/v3/services` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:32.566634 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-387fc687-96b6-4e08-a7b1-1e9dc5518af5 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:32.566934 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-387fc687-96b6-4e08-a7b1-1e9dc5518af5 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:32.568297 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-387fc687-96b6-4e08-a7b1-1e9dc5518af5 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:32.573323 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 437/869] 10.222.0.42 () {56 vars in 1126 bytes} [Sun Aug 28 00:42:32 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:32.579180 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b2b10d5-55e0-43ac-b37d-2bb99c62bdd5 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:32.628318 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:32.684404 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b2b10d5-55e0-43ac-b37d-2bb99c62bdd5 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:32.720861 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7064472-e448-44d0-98dc-64d71720436a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:32.721503 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7064472-e448-44d0-98dc-64d71720436a None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:32.721777 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7064472-e448-44d0-98dc-64d71720436a None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:32.722044 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7064472-e448-44d0-98dc-64d71720436a None admin] PATH_INFO: `/v3/services` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:32.722673 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7064472-e448-44d0-98dc-64d71720436a None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:32.722970 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7064472-e448-44d0-98dc-64d71720436a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:32.724339 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7064472-e448-44d0-98dc-64d71720436a None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:32.729573 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 433/870] 10.222.0.42 () {56 vars in 1126 bytes} [Sun Aug 28 00:42:32 2022] GET /identity/v3/services?type=compute_legacy => generated 131 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:32.792359 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ddb169-81d6-407b-8569-e0e0deefeed8 None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:32.792987 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ddb169-81d6-407b-8569-e0e0deefeed8 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:32.793272 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ddb169-81d6-407b-8569-e0e0deefeed8 None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:32.795039 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 438/871] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:42:32 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:42:32.801112 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e2b2e4a-17b0-43b2-96e4-b0b51a6b816f None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:32.801485 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e2b2e4a-17b0-43b2-96e4-b0b51a6b816f None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:32.801825 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e2b2e4a-17b0-43b2-96e4-b0b51a6b816f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:32.821182 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-0e2b2e4a-17b0-43b2-96e4-b0b51a6b816f None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:42:32.822414 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-0e2b2e4a-17b0-43b2-96e4-b0b51a6b816f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:42:32.898596 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 434/872] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:42:32 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 28 00:42:33.173246 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55ddb169-81d6-407b-8569-e0e0deefeed8 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:33.273909 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:33.285391 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55ddb169-81d6-407b-8569-e0e0deefeed8 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:33.321030 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51b08b6b-616c-4b48-b225-8978a2571160 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:33.321688 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51b08b6b-616c-4b48-b225-8978a2571160 None admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:33.321967 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51b08b6b-616c-4b48-b225-8978a2571160 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:33.322291 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51b08b6b-616c-4b48-b225-8978a2571160 None admin] PATH_INFO: `/v3/services` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:33.322963 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51b08b6b-616c-4b48-b225-8978a2571160 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:33.324341 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51b08b6b-616c-4b48-b225-8978a2571160 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:33.343523 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 439/873] 10.222.0.42 () {60 vars in 1141 bytes} [Sun Aug 28 00:42:33 2022] POST /identity/v3/services => generated 251 bytes in 171 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:42:33.895755 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67703d6e-e0a9-40a5-8c8c-403d45026af1 None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:33.896303 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67703d6e-e0a9-40a5-8c8c-403d45026af1 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:33.896303 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67703d6e-e0a9-40a5-8c8c-403d45026af1 None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:33.897948 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 435/874] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:42:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:42:33.903335 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e414585e-2873-421c-850a-34d32913a390 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:33.903787 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e414585e-2873-421c-850a-34d32913a390 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:33.903863 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e414585e-2873-421c-850a-34d32913a390 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:33.923667 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-e414585e-2873-421c-850a-34d32913a390 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:42:33.924878 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-e414585e-2873-421c-850a-34d32913a390 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:42:34.007009 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 440/875] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:42:33 2022] POST /identity/v3/auth/tokens => generated 1262 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 28 00:42:34.153452 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-175ba3da-7b27-4378-960a-87daa0bb5fa7 None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:34.153452 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-175ba3da-7b27-4378-960a-87daa0bb5fa7 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:34.153727 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-175ba3da-7b27-4378-960a-87daa0bb5fa7 None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:34.155243 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 436/876] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:42:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:42:34.160648 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8a237be-3db7-4dfa-8ce9-62f21eee9d26 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:34.160935 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8a237be-3db7-4dfa-8ce9-62f21eee9d26 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:34.161241 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8a237be-3db7-4dfa-8ce9-62f21eee9d26 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:34.179501 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-f8a237be-3db7-4dfa-8ce9-62f21eee9d26 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:42:34.180687 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-f8a237be-3db7-4dfa-8ce9-62f21eee9d26 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:42:34.253319 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 441/877] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:42:34 2022] POST /identity/v3/auth/tokens => generated 1262 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 28 00:42:34.268295 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-175ba3da-7b27-4378-960a-87daa0bb5fa7 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:34.311477 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:34.373352 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-175ba3da-7b27-4378-960a-87daa0bb5fa7 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:34.410409 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cce6951f-c645-48d7-9988-0b7ba5108566 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:34.411305 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cce6951f-c645-48d7-9988-0b7ba5108566 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:34.411732 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cce6951f-c645-48d7-9988-0b7ba5108566 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:34.412148 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cce6951f-c645-48d7-9988-0b7ba5108566 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:34.423601 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cce6951f-c645-48d7-9988-0b7ba5108566 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 28 00:42:34.428822 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-cce6951f-c645-48d7-9988-0b7ba5108566 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 28 00:42:34.428822 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:34.428822 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:34.428822 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:34.428822 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:34.428822 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:34.428822 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:34.428822 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:34.428822 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:34.428822 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:34.428822 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:34.428822 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:34.428822 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 28 00:42:34.428822 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 28 00:42:34.428822 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 28 00:42:34.428822 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 28 00:42:34.430001 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 28 00:42:34.430001 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 28 00:42:34.430001 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 28 00:42:34.430001 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 28 00:42:34.430001 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:34.430001 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:34.430001 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:34.430001 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:34.430001 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:34.430001 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:34.430001 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:34.430001 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:34.430001 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:34.430001 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:34.430001 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:34.430001 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:34.431100 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:34.431100 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:34.431100 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:34.431100 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:34.431100 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 28 00:42:34.431100 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 28 00:42:34.431100 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 28 00:42:34.431100 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 28 00:42:34.431100 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 28 00:42:34.431100 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 28 00:42:34.431100 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 28 00:42:34.431100 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:42:34.433509 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 437/878] 10.222.0.42 () {56 vars in 1120 bytes} [Sun Aug 28 00:42:34 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:34.445005 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8a237be-3db7-4dfa-8ce9-62f21eee9d26 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:34.461367 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:34.520677 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cce6951f-c645-48d7-9988-0b7ba5108566 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:34.544920 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8a237be-3db7-4dfa-8ce9-62f21eee9d26 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:34.584715 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3cfde3e-eba8-4500-8a0b-67969925e635 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:34.585807 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3cfde3e-eba8-4500-8a0b-67969925e635 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:34.586328 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3cfde3e-eba8-4500-8a0b-67969925e635 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:34.586815 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3cfde3e-eba8-4500-8a0b-67969925e635 None admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:34.587692 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3cfde3e-eba8-4500-8a0b-67969925e635 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:34.588208 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3cfde3e-eba8-4500-8a0b-67969925e635 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:34.589928 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3cfde3e-eba8-4500-8a0b-67969925e635 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:34.600177 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 442/879] 10.222.0.42 () {56 vars in 1115 bytes} [Sun Aug 28 00:42:34 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:34.605858 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3cfde3e-eba8-4500-8a0b-67969925e635 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:34.639934 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:34.643973 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cce6951f-c645-48d7-9988-0b7ba5108566 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:34.681451 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ed3d560-32f9-4a6a-bfec-ff39506b279b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:34.682008 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ed3d560-32f9-4a6a-bfec-ff39506b279b None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:34.682294 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ed3d560-32f9-4a6a-bfec-ff39506b279b None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:34.682567 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ed3d560-32f9-4a6a-bfec-ff39506b279b None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:34.683160 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ed3d560-32f9-4a6a-bfec-ff39506b279b None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:34.684557 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ed3d560-32f9-4a6a-bfec-ff39506b279b None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:34.689011 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:34.694116 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-8ed3d560-32f9-4a6a-bfec-ff39506b279b None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 28 00:42:34.694116 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:34.694116 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 28 00:42:34.694116 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 28 00:42:34.694116 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 28 00:42:34.694116 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 28 00:42:34.694116 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 28 00:42:34.694116 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 28 00:42:34.694116 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 28 00:42:34.694116 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:42:34.694116 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 28 00:42:34.694116 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:42:34.694116 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:34.694116 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:34.694116 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:34.694116 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:34.694116 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:34.695109 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:34.695109 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:34.695109 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:34.695109 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:34.695109 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:34.695109 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:34.695109 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 28 00:42:34.695109 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 28 00:42:34.695109 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 28 00:42:34.695109 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 28 00:42:34.695109 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:34.695109 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:34.695109 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:34.695109 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:34.695109 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:34.695109 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:34.695973 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:34.695973 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:34.695973 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:34.695973 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:34.695973 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:34.695973 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:34.695973 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:34.695973 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:34.695973 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:34.695973 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:34.695973 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 28 00:42:34.695973 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 28 00:42:34.695973 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 28 00:42:34.695973 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:42:34.695973 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 438/880] 10.222.0.42 () {56 vars in 1126 bytes} [Sun Aug 28 00:42:34 2022] GET /identity/v3/services/object-store => generated 93 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 28 00:42:34.702961 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ed3d560-32f9-4a6a-bfec-ff39506b279b None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:34.712627 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3cfde3e-eba8-4500-8a0b-67969925e635 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:34.749708 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-582874ff-19ad-429a-bb9a-b62dcf90ccd6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:34.750346 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-582874ff-19ad-429a-bb9a-b62dcf90ccd6 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:34.750625 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-582874ff-19ad-429a-bb9a-b62dcf90ccd6 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:34.750899 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-582874ff-19ad-429a-bb9a-b62dcf90ccd6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:34.764973 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-582874ff-19ad-429a-bb9a-b62dcf90ccd6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:34.766450 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-582874ff-19ad-429a-bb9a-b62dcf90ccd6 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:34.782457 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-582874ff-19ad-429a-bb9a-b62dcf90ccd6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:42:34.782457 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:34.782457 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:34.782457 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:34.782457 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:34.782457 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:34.782457 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:34.782457 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:34.782457 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:34.782457 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:34.782457 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:34.782457 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:34.782457 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 28 00:42:34.782457 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 28 00:42:34.782457 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 28 00:42:34.782457 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 28 00:42:34.783632 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:34.783632 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:34.783632 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:34.783632 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:34.783632 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:34.783632 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:34.783632 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:34.783632 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:34.783632 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:34.783632 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:34.783632 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:34.783632 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:34.783632 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:34.783632 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:34.783632 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:34.783632 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:34.783632 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 28 00:42:34.784723 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 28 00:42:34.784723 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:42:34.784723 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:34.788559 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 443/881] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:42:34 2022] GET /identity/v3/domains/Default => generated 87 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 28 00:42:34.789561 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-582874ff-19ad-429a-bb9a-b62dcf90ccd6 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:34.834105 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ed3d560-32f9-4a6a-bfec-ff39506b279b None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:34.881432 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5287d8c9-bd0b-464e-8eef-6bec64308754 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:34.882038 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5287d8c9-bd0b-464e-8eef-6bec64308754 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:34.882315 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5287d8c9-bd0b-464e-8eef-6bec64308754 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:34.882582 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5287d8c9-bd0b-464e-8eef-6bec64308754 None admin] PATH_INFO: `/v3/services` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:34.883206 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5287d8c9-bd0b-464e-8eef-6bec64308754 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:34.883891 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5287d8c9-bd0b-464e-8eef-6bec64308754 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:34.885307 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5287d8c9-bd0b-464e-8eef-6bec64308754 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:34.890341 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 439/882] 10.222.0.42 () {56 vars in 1122 bytes} [Sun Aug 28 00:42:34 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:34.891464 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ae383f-4f33-47e6-be8d-dd7e78629227 None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:34.891785 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ae383f-4f33-47e6-be8d-dd7e78629227 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:34.892059 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ae383f-4f33-47e6-be8d-dd7e78629227 None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:34.893442 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 440/883] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:42:34 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:42:34.897997 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66ae383f-4f33-47e6-be8d-dd7e78629227 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:34.901801 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-582874ff-19ad-429a-bb9a-b62dcf90ccd6 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:34.926136 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:34.929241 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:34.943801 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-489ca986-464c-442b-8057-2f140a73b13d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:34.944378 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-489ca986-464c-442b-8057-2f140a73b13d None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:34.944618 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-489ca986-464c-442b-8057-2f140a73b13d None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:34.944888 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-489ca986-464c-442b-8057-2f140a73b13d None admin] PATH_INFO: `/v3/domains` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:34.945555 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-489ca986-464c-442b-8057-2f140a73b13d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:34.945857 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-489ca986-464c-442b-8057-2f140a73b13d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:34.947240 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-489ca986-464c-442b-8057-2f140a73b13d None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:34.964659 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 444/884] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Aug 28 00:42:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:34.966085 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a09fb0-4469-4d57-afd4-2e91444533f1 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:34.966336 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a09fb0-4469-4d57-afd4-2e91444533f1 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:34.966606 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a09fb0-4469-4d57-afd4-2e91444533f1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:34.984650 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-53a09fb0-4469-4d57-afd4-2e91444533f1 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:42:34.986194 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-53a09fb0-4469-4d57-afd4-2e91444533f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:42:35.013939 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66ae383f-4f33-47e6-be8d-dd7e78629227 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:35.051802 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3539c8cc-be61-4b51-9b27-a16ec2cbafa6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:35.052392 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3539c8cc-be61-4b51-9b27-a16ec2cbafa6 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:35.052682 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3539c8cc-be61-4b51-9b27-a16ec2cbafa6 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:35.052933 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3539c8cc-be61-4b51-9b27-a16ec2cbafa6 None admin] PATH_INFO: `/v3/services` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:35.053576 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3539c8cc-be61-4b51-9b27-a16ec2cbafa6 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:35.053873 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3539c8cc-be61-4b51-9b27-a16ec2cbafa6 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:35.055234 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3539c8cc-be61-4b51-9b27-a16ec2cbafa6 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:35.059669 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 441/885] 10.222.0.42 () {56 vars in 1122 bytes} [Sun Aug 28 00:42:34 2022] GET /identity/v3/services?type=object-store => generated 129 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:35.060456 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3539c8cc-be61-4b51-9b27-a16ec2cbafa6 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:35.078416 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 445/886] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:42:34 2022] POST /identity/v3/auth/tokens => generated 1262 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 28 00:42:35.185560 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:35.195859 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3539c8cc-be61-4b51-9b27-a16ec2cbafa6 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:35.241838 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-089b83e1-9daa-4304-8c1d-33f0a63477c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:35.242694 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-089b83e1-9daa-4304-8c1d-33f0a63477c9 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:35.243182 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-089b83e1-9daa-4304-8c1d-33f0a63477c9 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:35.243531 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-089b83e1-9daa-4304-8c1d-33f0a63477c9 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:35.252219 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-089b83e1-9daa-4304-8c1d-33f0a63477c9 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:35.253724 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-089b83e1-9daa-4304-8c1d-33f0a63477c9 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:35.267590 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-089b83e1-9daa-4304-8c1d-33f0a63477c9 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 28 00:42:35.267590 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:35.267590 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:35.267590 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:35.267590 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:35.267590 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:35.267590 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:35.267590 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:35.267590 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:35.267590 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:35.267590 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:35.267590 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:35.267590 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 28 00:42:35.267590 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 28 00:42:35.267590 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 28 00:42:35.267590 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 28 00:42:35.268587 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:35.268587 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:35.268587 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 28 00:42:35.268587 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 28 00:42:35.268587 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 28 00:42:35.268587 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 28 00:42:35.268587 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:35.268587 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:35.268587 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:35.268587 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:35.268587 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:35.268587 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:35.268587 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:35.268587 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:35.268587 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:35.268587 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:35.268587 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:35.268587 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:35.270038 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:35.270038 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:35.270038 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 28 00:42:35.270038 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 28 00:42:35.270038 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 28 00:42:35.270038 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 28 00:42:35.270038 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 28 00:42:35.270038 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 28 00:42:35.270038 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 28 00:42:35.270038 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:42:35.270038 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 442/887] 10.222.0.42 () {56 vars in 1117 bytes} [Sun Aug 28 00:42:35 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:35.275699 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53a09fb0-4469-4d57-afd4-2e91444533f1 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:35.324482 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a30cc366-cfdd-4bc4-b4fa-05df199c4f9f None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:35.325037 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a30cc366-cfdd-4bc4-b4fa-05df199c4f9f None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:35.325572 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a30cc366-cfdd-4bc4-b4fa-05df199c4f9f None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:35.327400 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 443/888] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:42:35 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 28 00:42:35.333325 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-147ffef9-6510-4029-b8d2-ab6a53ab518d None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:35.333683 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-147ffef9-6510-4029-b8d2-ab6a53ab518d None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:35.334034 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-147ffef9-6510-4029-b8d2-ab6a53ab518d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:35.353605 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-147ffef9-6510-4029-b8d2-ab6a53ab518d None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:42:35.354809 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-147ffef9-6510-4029-b8d2-ab6a53ab518d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:42:35.389309 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53a09fb0-4469-4d57-afd4-2e91444533f1 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:35.404609 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:35.423017 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 444/889] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:42:35 2022] POST /identity/v3/auth/tokens => generated 1262 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 28 00:42:35.423845 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-147ffef9-6510-4029-b8d2-ab6a53ab518d None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:35.438214 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c67bb623-cfa9-49d4-9e5e-b0c9bf7174e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:35.439419 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c67bb623-cfa9-49d4-9e5e-b0c9bf7174e5 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:35.440386 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c67bb623-cfa9-49d4-9e5e-b0c9bf7174e5 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:35.441135 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c67bb623-cfa9-49d4-9e5e-b0c9bf7174e5 None admin] PATH_INFO: `/v3/users` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:35.442114 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c67bb623-cfa9-49d4-9e5e-b0c9bf7174e5 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:35.442854 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c67bb623-cfa9-49d4-9e5e-b0c9bf7174e5 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:35.444875 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c67bb623-cfa9-49d4-9e5e-b0c9bf7174e5 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:35.473663 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 446/890] 10.222.0.42 () {56 vars in 1149 bytes} [Sun Aug 28 00:42:35 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:35.474600 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c67bb623-cfa9-49d4-9e5e-b0c9bf7174e5 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:35.519737 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:35.544906 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-147ffef9-6510-4029-b8d2-ab6a53ab518d None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:35.581367 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72274bf7-84dd-427f-bc21-8b480890ded6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:35.582114 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72274bf7-84dd-427f-bc21-8b480890ded6 None admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:35.582454 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72274bf7-84dd-427f-bc21-8b480890ded6 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:35.582789 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72274bf7-84dd-427f-bc21-8b480890ded6 None admin] PATH_INFO: `/v3/services` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:35.583519 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72274bf7-84dd-427f-bc21-8b480890ded6 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:35.584936 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72274bf7-84dd-427f-bc21-8b480890ded6 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:35.604215 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c67bb623-cfa9-49d4-9e5e-b0c9bf7174e5 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:35.609468 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 445/891] 10.222.0.42 () {60 vars in 1141 bytes} [Sun Aug 28 00:42:35 2022] POST /identity/v3/services => generated 269 bytes in 186 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:42:35.650955 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8005f35a-2161-4121-843d-fb2bd5df7b59 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:35.651985 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8005f35a-2161-4121-843d-fb2bd5df7b59 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:35.652470 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8005f35a-2161-4121-843d-fb2bd5df7b59 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:35.652941 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8005f35a-2161-4121-843d-fb2bd5df7b59 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:35.691277 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72274bf7-84dd-427f-bc21-8b480890ded6 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:35.692128 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:35.730260 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8005f35a-2161-4121-843d-fb2bd5df7b59 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:35.731689 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8005f35a-2161-4121-843d-fb2bd5df7b59 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:35.827343 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72274bf7-84dd-427f-bc21-8b480890ded6 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:35.866648 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d64a091-fa28-43f8-81a5-2fecb0fc8700 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:35.867371 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d64a091-fa28-43f8-81a5-2fecb0fc8700 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:35.867639 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 447/892] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Aug 28 00:42:35 2022] GET /identity/v3/auth/tokens => generated 1372 bytes in 393 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 28 00:42:35.867791 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d64a091-fa28-43f8-81a5-2fecb0fc8700 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:35.868069 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d64a091-fa28-43f8-81a5-2fecb0fc8700 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:35.868662 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d64a091-fa28-43f8-81a5-2fecb0fc8700 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:35.870149 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d64a091-fa28-43f8-81a5-2fecb0fc8700 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:35.877405 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8005f35a-2161-4121-843d-fb2bd5df7b59 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:35.880223 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-8d64a091-fa28-43f8-81a5-2fecb0fc8700 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 28 00:42:35.880223 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:35.880223 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 28 00:42:35.880223 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 28 00:42:35.880223 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 28 00:42:35.880223 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 28 00:42:35.880223 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 28 00:42:35.880223 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 28 00:42:35.880223 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 28 00:42:35.880223 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:42:35.880223 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 28 00:42:35.880223 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:42:35.880223 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:35.880223 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:35.880223 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:35.880223 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:35.880223 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:35.881230 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:35.881230 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:35.881230 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:35.881230 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:35.881230 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:35.881230 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:35.881230 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 28 00:42:35.881230 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 28 00:42:35.881230 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 28 00:42:35.881230 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 28 00:42:35.881230 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:35.881230 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:35.881230 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:35.881230 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:35.881230 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:35.881230 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:35.882066 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:35.882066 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:35.882066 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:35.882066 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:35.882066 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:35.882066 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:35.882066 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:35.882066 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:35.882066 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:35.882066 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:35.882066 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 28 00:42:35.882066 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 28 00:42:35.882066 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 28 00:42:35.882066 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:42:35.884741 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 446/893] 10.222.0.42 () {56 vars in 1114 bytes} [Sun Aug 28 00:42:35 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 28 00:42:35.893467 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d64a091-fa28-43f8-81a5-2fecb0fc8700 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:35.906463 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:35.966506 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:36.012257 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8005f35a-2161-4121-843d-fb2bd5df7b59 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:36.033235 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d64a091-fa28-43f8-81a5-2fecb0fc8700 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:36.058573 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1b411e4-4bc2-491a-bbdc-67099acdf8a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:36.059557 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1b411e4-4bc2-491a-bbdc-67099acdf8a0 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:36.060018 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1b411e4-4bc2-491a-bbdc-67099acdf8a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:36.060457 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1b411e4-4bc2-491a-bbdc-67099acdf8a0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:36.075496 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1b411e4-4bc2-491a-bbdc-67099acdf8a0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:36.077383 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1b411e4-4bc2-491a-bbdc-67099acdf8a0 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:36.078427 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2779bca5-25f6-4bcf-ae4e-0a4ef37125ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:36.079147 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2779bca5-25f6-4bcf-ae4e-0a4ef37125ed None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:36.079524 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2779bca5-25f6-4bcf-ae4e-0a4ef37125ed None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:36.080509 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2779bca5-25f6-4bcf-ae4e-0a4ef37125ed None admin] PATH_INFO: `/v3/services` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:36.082299 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2779bca5-25f6-4bcf-ae4e-0a4ef37125ed None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:36.082299 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2779bca5-25f6-4bcf-ae4e-0a4ef37125ed None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:36.083611 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2779bca5-25f6-4bcf-ae4e-0a4ef37125ed None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:36.097741 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 447/894] 10.222.0.42 () {56 vars in 1114 bytes} [Sun Aug 28 00:42:35 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:36.098232 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-b1b411e4-4bc2-491a-bbdc-67099acdf8a0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:42:36.098232 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:36.098232 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:36.098232 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:36.098232 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:36.098232 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:36.098232 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:36.098232 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:36.098232 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:36.098232 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:36.098232 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:36.098232 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:36.098232 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 28 00:42:36.098232 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 28 00:42:36.098232 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 28 00:42:36.098232 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 28 00:42:36.099315 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:36.099315 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:36.099315 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:36.099315 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:36.099315 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:36.099315 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:36.099315 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:36.099315 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:36.099315 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:36.099315 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:36.099315 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:36.099315 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:36.099315 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:36.099315 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:36.099315 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:36.099315 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:36.099315 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 28 00:42:36.100219 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 28 00:42:36.100219 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:42:36.100219 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:36.100397 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2779bca5-25f6-4bcf-ae4e-0a4ef37125ed None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:36.106985 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 448/895] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:42:35 2022] GET /identity/v3/domains/Default => generated 87 bytes in 225 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:36.108779 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1b411e4-4bc2-491a-bbdc-67099acdf8a0 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:36.213107 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:36.231885 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1b411e4-4bc2-491a-bbdc-67099acdf8a0 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:36.251560 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:36.264502 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2779bca5-25f6-4bcf-ae4e-0a4ef37125ed None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:36.270663 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73079b4b-ff45-4277-847a-c48209965d42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:36.271275 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73079b4b-ff45-4277-847a-c48209965d42 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:36.271550 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73079b4b-ff45-4277-847a-c48209965d42 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:36.271817 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73079b4b-ff45-4277-847a-c48209965d42 None admin] PATH_INFO: `/v3/domains` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:36.272453 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73079b4b-ff45-4277-847a-c48209965d42 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:36.272747 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73079b4b-ff45-4277-847a-c48209965d42 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:36.274172 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73079b4b-ff45-4277-847a-c48209965d42 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:36.291616 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 449/896] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Aug 28 00:42:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:36.292585 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73079b4b-ff45-4277-847a-c48209965d42 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:36.307591 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eac45fa3-5c49-42ae-a4b0-9ec4c53f70ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:36.308248 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac45fa3-5c49-42ae-a4b0-9ec4c53f70ff None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:36.308529 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac45fa3-5c49-42ae-a4b0-9ec4c53f70ff None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:36.308794 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac45fa3-5c49-42ae-a4b0-9ec4c53f70ff None admin] PATH_INFO: `/v3/services` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:36.309478 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eac45fa3-5c49-42ae-a4b0-9ec4c53f70ff None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:36.309733 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eac45fa3-5c49-42ae-a4b0-9ec4c53f70ff None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:36.311170 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eac45fa3-5c49-42ae-a4b0-9ec4c53f70ff None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:36.321011 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 448/897] 10.222.0.42 () {56 vars in 1114 bytes} [Sun Aug 28 00:42:36 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:36.322758 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eac45fa3-5c49-42ae-a4b0-9ec4c53f70ff None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:36.425391 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73079b4b-ff45-4277-847a-c48209965d42 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:36.445058 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eac45fa3-5c49-42ae-a4b0-9ec4c53f70ff None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:36.481868 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-387db940-9e6f-4b21-9ce6-67d6f67943a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:36.482935 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-387db940-9e6f-4b21-9ce6-67d6f67943a7 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:36.483420 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-387db940-9e6f-4b21-9ce6-67d6f67943a7 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:36.484120 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-387db940-9e6f-4b21-9ce6-67d6f67943a7 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:36.485091 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6e4926f-24dc-400a-bbf4-5ab76b42dc2b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:36.485841 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6e4926f-24dc-400a-bbf4-5ab76b42dc2b None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:36.486118 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6e4926f-24dc-400a-bbf4-5ab76b42dc2b None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:36.486387 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6e4926f-24dc-400a-bbf4-5ab76b42dc2b None admin] PATH_INFO: `/v3/endpoints` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:36.487233 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6e4926f-24dc-400a-bbf4-5ab76b42dc2b None admin] RBAC: Adding query filter params (interface=public, service_id=86a4c71b91404db8ad4f6ea94dee7632, region_id=RegionOne) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:36.487532 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6e4926f-24dc-400a-bbf4-5ab76b42dc2b None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:36.489153 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6e4926f-24dc-400a-bbf4-5ab76b42dc2b None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:36.491060 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:36.496365 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-387db940-9e6f-4b21-9ce6-67d6f67943a7 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:36.498742 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-387db940-9e6f-4b21-9ce6-67d6f67943a7 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:36.504254 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 449/898] 10.222.0.42 () {56 vars in 1251 bytes} [Sun Aug 28 00:42:36 2022] GET /identity/v3/endpoints?service_id=86a4c71b91404db8ad4f6ea94dee7632&interface=public®ion_id=RegionOne => generated 194 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 28 00:42:36.505254 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6e4926f-24dc-400a-bbf4-5ab76b42dc2b None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:36.516830 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-387db940-9e6f-4b21-9ce6-67d6f67943a7 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 28 00:42:36.516830 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:36.516830 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:36.516830 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:36.516830 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:36.516830 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:36.516830 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:36.516830 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:36.516830 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:36.516830 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:36.516830 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:36.516830 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:36.516830 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 28 00:42:36.516830 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 28 00:42:36.516830 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 28 00:42:36.518179 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 28 00:42:36.518179 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:36.518179 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:36.518179 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:36.518179 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:36.518179 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:36.518179 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:36.518179 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:36.518179 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:36.518179 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:36.518179 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:36.518179 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:36.518179 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:36.518179 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:36.518179 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:36.518179 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:36.518179 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:36.519221 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 28 00:42:36.519221 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 28 00:42:36.519221 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 28 00:42:36.519221 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 28 00:42:36.519221 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 28 00:42:36.519221 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 28 00:42:36.519221 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 28 00:42:36.519221 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:36.522445 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 450/899] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Aug 28 00:42:36 2022] GET /identity/v3/projects/service => generated 88 bytes in 230 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:36.527793 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-387db940-9e6f-4b21-9ce6-67d6f67943a7 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:36.588739 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:36.662630 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6e4926f-24dc-400a-bbf4-5ab76b42dc2b None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:36.671985 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-387db940-9e6f-4b21-9ce6-67d6f67943a7 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:36.711254 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bfea59e-aa68-4d39-8675-147c6a7195d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:36.712310 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bfea59e-aa68-4d39-8675-147c6a7195d2 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:36.712955 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bfea59e-aa68-4d39-8675-147c6a7195d2 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:36.713432 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bfea59e-aa68-4d39-8675-147c6a7195d2 None admin] PATH_INFO: `/v3/services` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:36.714210 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bfea59e-aa68-4d39-8675-147c6a7195d2 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:36.714316 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-690b413d-7414-46c5-8b5d-2665bd21c421 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:36.714966 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-690b413d-7414-46c5-8b5d-2665bd21c421 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:36.715317 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-690b413d-7414-46c5-8b5d-2665bd21c421 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:36.715653 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-690b413d-7414-46c5-8b5d-2665bd21c421 None admin] PATH_INFO: `/v3/projects` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:36.716423 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-690b413d-7414-46c5-8b5d-2665bd21c421 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:36.716661 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bfea59e-aa68-4d39-8675-147c6a7195d2 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:36.716774 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-690b413d-7414-46c5-8b5d-2665bd21c421 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:36.718703 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-690b413d-7414-46c5-8b5d-2665bd21c421 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:36.730195 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 450/900] 10.222.0.42 () {56 vars in 1087 bytes} [Sun Aug 28 00:42:36 2022] GET /identity/v3/services => generated 1268 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 00:42:36.743175 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 451/901] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Aug 28 00:42:36 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:36.749574 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bfea59e-aa68-4d39-8675-147c6a7195d2 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:36.825044 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:36.868971 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bfea59e-aa68-4d39-8675-147c6a7195d2 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:36.915886 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae3fca7e-ae13-4156-a1b5-dd55969e18a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:36.916656 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae3fca7e-ae13-4156-a1b5-dd55969e18a0 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:36.917040 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae3fca7e-ae13-4156-a1b5-dd55969e18a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:36.917294 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae3fca7e-ae13-4156-a1b5-dd55969e18a0 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:36.918023 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae3fca7e-ae13-4156-a1b5-dd55969e18a0 None admin] RBAC: Adding query filter params (role.id=2c6107d249594bd3abf791034b71e98f, scope.project.id=f53e6fdb369b480fbebb4086a505df67, user.id=eede08f020e04282bb994b855e371433) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:36.918329 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae3fca7e-ae13-4156-a1b5-dd55969e18a0 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:36.919830 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae3fca7e-ae13-4156-a1b5-dd55969e18a0 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:36.929539 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 451/902] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Aug 28 00:42:36 2022] GET /identity/v3/role_assignments?user.id=eede08f020e04282bb994b855e371433&scope.project.id=f53e6fdb369b480fbebb4086a505df67&role.id=2c6107d249594bd3abf791034b71e98f => generated 259 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:37.091997 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea279d81-d168-4a18-a96a-db85e37287ee None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:37.092499 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea279d81-d168-4a18-a96a-db85e37287ee None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:37.092702 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea279d81-d168-4a18-a96a-db85e37287ee None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:37.094549 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 452/903] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:42:37 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:42:37.101171 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c6d5ce7-f806-4bca-9246-cc8459d19589 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:37.101538 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c6d5ce7-f806-4bca-9246-cc8459d19589 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:37.101818 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c6d5ce7-f806-4bca-9246-cc8459d19589 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:37.128742 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-4c6d5ce7-f806-4bca-9246-cc8459d19589 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:42:37.130299 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-4c6d5ce7-f806-4bca-9246-cc8459d19589 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:42:37.205836 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 452/904] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:42:37 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 28 00:42:37.477754 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea279d81-d168-4a18-a96a-db85e37287ee None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:37.588279 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28688902-be24-4d8c-83c0-d18c849e9f92 None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:37.588567 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28688902-be24-4d8c-83c0-d18c849e9f92 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:37.588859 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28688902-be24-4d8c-83c0-d18c849e9f92 None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:37.590422 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 453/905] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:42:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:42:37.596442 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1975d624-840e-41a5-ac5b-156efc160f8a None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:37.596442 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1975d624-840e-41a5-ac5b-156efc160f8a None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:37.596699 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1975d624-840e-41a5-ac5b-156efc160f8a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:37.606258 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:37.608760 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea279d81-d168-4a18-a96a-db85e37287ee None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:37.617896 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-1975d624-840e-41a5-ac5b-156efc160f8a None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:42:37.618882 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-1975d624-840e-41a5-ac5b-156efc160f8a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:42:37.654986 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4501f0c-813f-4b1f-8526-3cfb3d9c4036 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:37.656577 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4501f0c-813f-4b1f-8526-3cfb3d9c4036 None admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:37.657096 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4501f0c-813f-4b1f-8526-3cfb3d9c4036 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:37.657602 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4501f0c-813f-4b1f-8526-3cfb3d9c4036 None admin] PATH_INFO: `/v3/services` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:37.658455 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4501f0c-813f-4b1f-8526-3cfb3d9c4036 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:37.660018 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4501f0c-813f-4b1f-8526-3cfb3d9c4036 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:37.680472 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:37.682422 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 453/906] 10.222.0.42 () {60 vars in 1141 bytes} [Sun Aug 28 00:42:37 2022] POST /identity/v3/services => generated 241 bytes in 205 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:42:37.713638 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 454/907] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:42:37 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 28 00:42:37.983142 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4501f0c-813f-4b1f-8526-3cfb3d9c4036 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:38.094367 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4501f0c-813f-4b1f-8526-3cfb3d9c4036 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:38.145976 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e87364d-1b12-4f35-b48e-7861b3d40b00 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:38.147061 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e87364d-1b12-4f35-b48e-7861b3d40b00 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:38.147562 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e87364d-1b12-4f35-b48e-7861b3d40b00 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:38.148025 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e87364d-1b12-4f35-b48e-7861b3d40b00 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:38.148824 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e87364d-1b12-4f35-b48e-7861b3d40b00 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:38.150431 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e87364d-1b12-4f35-b48e-7861b3d40b00 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:38.166377 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-6e87364d-1b12-4f35-b48e-7861b3d40b00 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 28 00:42:38.166377 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:38.166377 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 28 00:42:38.166377 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 28 00:42:38.166377 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 28 00:42:38.166377 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 28 00:42:38.166377 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 28 00:42:38.166377 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 28 00:42:38.166377 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 28 00:42:38.166377 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:38.166377 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 28 00:42:38.166377 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:38.166377 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:38.166377 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:38.166377 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:38.166377 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:38.167418 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:38.167418 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:38.167418 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:38.167418 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:38.167418 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:38.167418 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:38.167418 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:38.167418 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 28 00:42:38.167418 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 28 00:42:38.167418 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 28 00:42:38.167418 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 28 00:42:38.167418 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:38.167418 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:38.167418 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:38.167418 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:38.167418 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:38.167418 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:38.168324 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:38.168324 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:38.168324 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:38.168324 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:38.168324 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:38.168324 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:38.168324 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:38.168324 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:38.168324 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:38.168324 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:38.168324 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 28 00:42:38.168324 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 28 00:42:38.168324 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 28 00:42:38.168324 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:38.171478 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 454/908] 10.222.0.42 () {56 vars in 1132 bytes} [Sun Aug 28 00:42:37 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:38.177083 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1975d624-840e-41a5-ac5b-156efc160f8a None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:38.284328 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1975d624-840e-41a5-ac5b-156efc160f8a None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:38.322434 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3b0501b-1cad-424a-974b-d77f1a945941 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:38.323048 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3b0501b-1cad-424a-974b-d77f1a945941 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:38.323325 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3b0501b-1cad-424a-974b-d77f1a945941 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:38.323589 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3b0501b-1cad-424a-974b-d77f1a945941 None admin] PATH_INFO: `/v3/services` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:38.324224 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3b0501b-1cad-424a-974b-d77f1a945941 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:38.324517 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3b0501b-1cad-424a-974b-d77f1a945941 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:38.325943 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3b0501b-1cad-424a-974b-d77f1a945941 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:38.332425 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 455/909] 10.222.0.42 () {56 vars in 1126 bytes} [Sun Aug 28 00:42:38 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:38.338338 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e87364d-1b12-4f35-b48e-7861b3d40b00 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:38.366358 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:38.448037 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e87364d-1b12-4f35-b48e-7861b3d40b00 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:38.485771 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88a18f68-8f8a-4659-ac55-d21a028f07cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:38.486916 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88a18f68-8f8a-4659-ac55-d21a028f07cd None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:38.487638 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88a18f68-8f8a-4659-ac55-d21a028f07cd None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:38.488344 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88a18f68-8f8a-4659-ac55-d21a028f07cd None admin] PATH_INFO: `/v3/services` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:38.489629 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88a18f68-8f8a-4659-ac55-d21a028f07cd None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:38.490184 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88a18f68-8f8a-4659-ac55-d21a028f07cd None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:38.492559 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88a18f68-8f8a-4659-ac55-d21a028f07cd None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:38.500679 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 455/910] 10.222.0.42 () {56 vars in 1126 bytes} [Sun Aug 28 00:42:38 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:38.506270 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3b0501b-1cad-424a-974b-d77f1a945941 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:38.557487 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:38.615479 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3b0501b-1cad-424a-974b-d77f1a945941 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:38.651854 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8b0d65c-e819-4df5-82f0-15283406615d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:38.652474 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b0d65c-e819-4df5-82f0-15283406615d None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:38.652748 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b0d65c-e819-4df5-82f0-15283406615d None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:38.653040 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b0d65c-e819-4df5-82f0-15283406615d None admin] PATH_INFO: `/v3/endpoints` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:38.653740 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8b0d65c-e819-4df5-82f0-15283406615d None admin] RBAC: Adding query filter params (interface=public, service_id=c06e34631b6b4088961ac77f6aabb672, region_id=RegionOne) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:38.654040 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8b0d65c-e819-4df5-82f0-15283406615d None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:38.655489 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8b0d65c-e819-4df5-82f0-15283406615d None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:38.661519 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 456/911] 10.222.0.42 () {56 vars in 1251 bytes} [Sun Aug 28 00:42:38 2022] GET /identity/v3/endpoints?service_id=c06e34631b6b4088961ac77f6aabb672&interface=public®ion_id=RegionOne => generated 194 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 28 00:42:38.667164 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88a18f68-8f8a-4659-ac55-d21a028f07cd None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:38.765336 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:38.787137 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88a18f68-8f8a-4659-ac55-d21a028f07cd None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:38.827396 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-062ae945-cc66-492e-a5d6-99fcdb25790c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:38.827989 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-062ae945-cc66-492e-a5d6-99fcdb25790c None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:38.828263 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-062ae945-cc66-492e-a5d6-99fcdb25790c None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:38.828529 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-062ae945-cc66-492e-a5d6-99fcdb25790c None admin] PATH_INFO: `/v3/services` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:38.829215 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-062ae945-cc66-492e-a5d6-99fcdb25790c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:38.830907 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-062ae945-cc66-492e-a5d6-99fcdb25790c None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:38.838867 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 456/912] 10.222.0.42 () {56 vars in 1087 bytes} [Sun Aug 28 00:42:38 2022] GET /identity/v3/services => generated 1497 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 00:42:38.878889 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b3c273e-39fe-4a91-9471-f6787fc2cd1d None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:38.879248 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b3c273e-39fe-4a91-9471-f6787fc2cd1d None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:38.879433 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b3c273e-39fe-4a91-9471-f6787fc2cd1d None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:38.881132 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 457/913] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:42:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:42:38.886964 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89643f7c-8ee5-46fe-9804-9155ff70413c None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:38.887311 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89643f7c-8ee5-46fe-9804-9155ff70413c None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:38.887524 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89643f7c-8ee5-46fe-9804-9155ff70413c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:38.887916 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40b2aecc-97ca-4d26-bcfb-6966c972486b None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:38.888208 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40b2aecc-97ca-4d26-bcfb-6966c972486b None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:38.888484 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40b2aecc-97ca-4d26-bcfb-6966c972486b None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:38.890332 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 458/914] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:42:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:42:38.897749 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9fbe5fc-a2fc-48b2-8ffc-90e8758a8b95 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:38.898049 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9fbe5fc-a2fc-48b2-8ffc-90e8758a8b95 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:38.898367 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9fbe5fc-a2fc-48b2-8ffc-90e8758a8b95 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:38.909552 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-89643f7c-8ee5-46fe-9804-9155ff70413c None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:42:38.910809 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-89643f7c-8ee5-46fe-9804-9155ff70413c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:42:38.927132 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-a9fbe5fc-a2fc-48b2-8ffc-90e8758a8b95 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:42:38.929058 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-a9fbe5fc-a2fc-48b2-8ffc-90e8758a8b95 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:42:39.003326 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 457/915] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:42:38 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 28 00:42:39.027579 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 459/916] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:42:38 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 28 00:42:39.277965 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89643f7c-8ee5-46fe-9804-9155ff70413c None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:39.296172 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9fbe5fc-a2fc-48b2-8ffc-90e8758a8b95 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:39.370284 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:39.401406 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:39.409662 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89643f7c-8ee5-46fe-9804-9155ff70413c None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:39.452238 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d425a644-4205-42d0-9e78-5e185135ceac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:39.453138 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d425a644-4205-42d0-9e78-5e185135ceac None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:39.453425 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d425a644-4205-42d0-9e78-5e185135ceac None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:39.453692 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d425a644-4205-42d0-9e78-5e185135ceac None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:39.467094 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d425a644-4205-42d0-9e78-5e185135ceac None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 28 00:42:39.473198 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-d425a644-4205-42d0-9e78-5e185135ceac None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 28 00:42:39.473198 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:39.473198 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:39.473198 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:39.473198 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:39.473198 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:39.473198 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:39.473198 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:39.473198 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:39.473198 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:39.473198 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:39.473198 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:39.473198 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 28 00:42:39.473198 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 28 00:42:39.473198 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 28 00:42:39.473198 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 28 00:42:39.475102 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 28 00:42:39.475102 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 28 00:42:39.475102 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 28 00:42:39.475102 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 28 00:42:39.475102 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:39.475102 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:39.475102 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:39.475102 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:39.475102 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:39.475102 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:39.475102 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:39.475102 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:39.475102 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:39.475102 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:39.475102 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:39.475102 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:39.477033 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:39.477033 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:39.477033 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:39.477033 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:39.477033 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 28 00:42:39.477033 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 28 00:42:39.477033 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 28 00:42:39.477033 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 28 00:42:39.477033 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 28 00:42:39.477033 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 28 00:42:39.477033 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 28 00:42:39.477033 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:39.479392 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 458/917] 10.222.0.42 () {56 vars in 1120 bytes} [Sun Aug 28 00:42:39 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:39.480660 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d425a644-4205-42d0-9e78-5e185135ceac None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:39.496203 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9fbe5fc-a2fc-48b2-8ffc-90e8758a8b95 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:39.556461 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5161f390-d5b7-4ba1-b24d-b672fccc36b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:39.558100 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5161f390-d5b7-4ba1-b24d-b672fccc36b4 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:39.558804 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5161f390-d5b7-4ba1-b24d-b672fccc36b4 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:39.559477 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5161f390-d5b7-4ba1-b24d-b672fccc36b4 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:39.560767 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5161f390-d5b7-4ba1-b24d-b672fccc36b4 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:39.563673 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5161f390-d5b7-4ba1-b24d-b672fccc36b4 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:39.579921 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-5161f390-d5b7-4ba1-b24d-b672fccc36b4 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 28 00:42:39.579921 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:39.579921 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 28 00:42:39.579921 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 28 00:42:39.579921 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 28 00:42:39.579921 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 28 00:42:39.579921 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 28 00:42:39.579921 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 28 00:42:39.579921 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 28 00:42:39.579921 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:42:39.579921 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 28 00:42:39.579921 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:42:39.579921 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:39.579921 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:39.579921 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:39.579921 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:39.579921 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:39.581920 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:39.581920 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:39.581920 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:39.581920 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:39.581920 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:39.581920 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:39.581920 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 28 00:42:39.581920 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 28 00:42:39.581920 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 28 00:42:39.581920 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 28 00:42:39.581920 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:39.581920 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:39.581920 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:39.581920 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:39.581920 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:39.581920 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:39.583410 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:39.583410 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:39.583410 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:39.583410 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:39.583410 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:39.583410 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:39.583410 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:39.583410 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:39.583410 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:39.583410 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:39.583410 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 28 00:42:39.583410 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 28 00:42:39.583410 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Aug 28 00:42:39.583410 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:42:39.587706 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 460/918] 10.222.0.42 () {56 vars in 1114 bytes} [Sun Aug 28 00:42:39 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 292 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 28 00:42:39.594278 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5161f390-d5b7-4ba1-b24d-b672fccc36b4 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:39.604509 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d425a644-4205-42d0-9e78-5e185135ceac None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:39.645906 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e35a197e-d6dc-40b1-8961-3cc0d1b9c16b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:39.646536 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e35a197e-d6dc-40b1-8961-3cc0d1b9c16b None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:39.646814 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e35a197e-d6dc-40b1-8961-3cc0d1b9c16b None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:39.647082 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e35a197e-d6dc-40b1-8961-3cc0d1b9c16b None admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:39.647766 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e35a197e-d6dc-40b1-8961-3cc0d1b9c16b None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:39.648074 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e35a197e-d6dc-40b1-8961-3cc0d1b9c16b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:39.649550 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e35a197e-d6dc-40b1-8961-3cc0d1b9c16b None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:39.658991 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 459/919] 10.222.0.42 () {56 vars in 1115 bytes} [Sun Aug 28 00:42:39 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:39.660036 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b78cc8b-ecc7-42b7-a385-db7ea726fe9c None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:39.660328 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b78cc8b-ecc7-42b7-a385-db7ea726fe9c None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:39.660598 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b78cc8b-ecc7-42b7-a385-db7ea726fe9c None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:39.666808 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 460/920] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:42:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:42:39.666808 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b78cc8b-ecc7-42b7-a385-db7ea726fe9c None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:39.668451 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:39.725732 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5161f390-d5b7-4ba1-b24d-b672fccc36b4 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:39.728785 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:39.763624 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2beb6b36-1999-452d-9bb0-8890bbc99d27 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:39.764394 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2beb6b36-1999-452d-9bb0-8890bbc99d27 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:39.764761 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2beb6b36-1999-452d-9bb0-8890bbc99d27 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:39.765072 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2beb6b36-1999-452d-9bb0-8890bbc99d27 None admin] PATH_INFO: `/v3/services` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:39.765801 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2beb6b36-1999-452d-9bb0-8890bbc99d27 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:39.766101 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2beb6b36-1999-452d-9bb0-8890bbc99d27 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:39.767499 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2beb6b36-1999-452d-9bb0-8890bbc99d27 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:39.776945 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 461/921] 10.222.0.42 () {56 vars in 1114 bytes} [Sun Aug 28 00:42:39 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:39.778415 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b78cc8b-ecc7-42b7-a385-db7ea726fe9c None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:39.778603 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-288763c3-f5fd-4106-a0a4-94147e68ca8e None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:39.778951 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-288763c3-f5fd-4106-a0a4-94147e68ca8e None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:39.779228 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-288763c3-f5fd-4106-a0a4-94147e68ca8e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:39.797890 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-288763c3-f5fd-4106-a0a4-94147e68ca8e None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:42:39.799221 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-288763c3-f5fd-4106-a0a4-94147e68ca8e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:42:39.818714 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-595b14b8-5f42-418c-b38d-f0c80094bb2c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:39.819696 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-595b14b8-5f42-418c-b38d-f0c80094bb2c None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:39.820621 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-595b14b8-5f42-418c-b38d-f0c80094bb2c None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:39.821124 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-595b14b8-5f42-418c-b38d-f0c80094bb2c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:39.832573 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-595b14b8-5f42-418c-b38d-f0c80094bb2c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:39.834413 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-595b14b8-5f42-418c-b38d-f0c80094bb2c None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:39.850926 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-595b14b8-5f42-418c-b38d-f0c80094bb2c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:42:39.850926 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:39.850926 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:39.850926 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:39.850926 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:39.850926 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:39.850926 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:39.850926 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:39.850926 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:39.850926 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:39.850926 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:39.850926 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:39.850926 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 28 00:42:39.850926 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 28 00:42:39.850926 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 28 00:42:39.850926 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 28 00:42:39.851923 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:39.851923 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:39.851923 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:39.851923 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:39.851923 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:39.851923 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:39.851923 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:39.851923 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:39.851923 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:39.851923 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:39.851923 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:39.851923 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:39.851923 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:39.851923 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:39.851923 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:39.851923 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:39.851923 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 28 00:42:39.852896 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 28 00:42:39.852896 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:42:39.852896 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:39.854669 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 461/922] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:42:39 2022] GET /identity/v3/domains/Default => generated 87 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:39.855755 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-595b14b8-5f42-418c-b38d-f0c80094bb2c None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:39.876163 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 462/923] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:42:39 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 28 00:42:39.877071 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-288763c3-f5fd-4106-a0a4-94147e68ca8e None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:39.925872 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:39.975769 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-595b14b8-5f42-418c-b38d-f0c80094bb2c None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:40.005667 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:40.010501 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-288763c3-f5fd-4106-a0a4-94147e68ca8e None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:40.016082 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeb170b3-035b-458d-9fbe-b3a2dc367549 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:40.017060 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeb170b3-035b-458d-9fbe-b3a2dc367549 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:40.017654 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeb170b3-035b-458d-9fbe-b3a2dc367549 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:40.018120 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeb170b3-035b-458d-9fbe-b3a2dc367549 None admin] PATH_INFO: `/v3/services` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:40.018989 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eeb170b3-035b-458d-9fbe-b3a2dc367549 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:40.019519 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eeb170b3-035b-458d-9fbe-b3a2dc367549 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:40.021241 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eeb170b3-035b-458d-9fbe-b3a2dc367549 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:40.027397 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 462/924] 10.222.0.42 () {56 vars in 1114 bytes} [Sun Aug 28 00:42:39 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:40.032807 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeb170b3-035b-458d-9fbe-b3a2dc367549 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:40.055614 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-802721c4-8bc1-4848-b053-d36eeaf75bfa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:40.056539 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-802721c4-8bc1-4848-b053-d36eeaf75bfa None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:40.057039 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-802721c4-8bc1-4848-b053-d36eeaf75bfa None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:40.057516 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-802721c4-8bc1-4848-b053-d36eeaf75bfa None admin] PATH_INFO: `/v3/domains` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:40.058350 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-802721c4-8bc1-4848-b053-d36eeaf75bfa None admin] RBAC: Adding query filter params (name=Default) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:40.058833 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-802721c4-8bc1-4848-b053-d36eeaf75bfa None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:40.060458 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-802721c4-8bc1-4848-b053-d36eeaf75bfa None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:40.074472 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 463/925] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Aug 28 00:42:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:40.086487 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-802721c4-8bc1-4848-b053-d36eeaf75bfa None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:40.154001 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeb170b3-035b-458d-9fbe-b3a2dc367549 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:40.172565 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:40.191307 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eac9399-0e1e-4445-b379-0d73f8cdf46d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:40.191850 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eac9399-0e1e-4445-b379-0d73f8cdf46d None admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:40.192129 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eac9399-0e1e-4445-b379-0d73f8cdf46d None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:40.192402 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eac9399-0e1e-4445-b379-0d73f8cdf46d None admin] PATH_INFO: `/v3/endpoints` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:40.193123 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eac9399-0e1e-4445-b379-0d73f8cdf46d None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:40.194537 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eac9399-0e1e-4445-b379-0d73f8cdf46d None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:40.205579 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-802721c4-8bc1-4848-b053-d36eeaf75bfa None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:40.221026 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 463/926] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 00:42:40 2022] POST /identity/v3/endpoints => generated 346 bytes in 188 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:42:40.221459 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eac9399-0e1e-4445-b379-0d73f8cdf46d None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:40.261497 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3c40b1c-0e01-4df1-b377-4c6bef856d64 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:40.262526 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3c40b1c-0e01-4df1-b377-4c6bef856d64 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:40.263066 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3c40b1c-0e01-4df1-b377-4c6bef856d64 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:40.263535 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3c40b1c-0e01-4df1-b377-4c6bef856d64 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:40.272819 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3c40b1c-0e01-4df1-b377-4c6bef856d64 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:40.274703 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3c40b1c-0e01-4df1-b377-4c6bef856d64 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:40.288693 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-a3c40b1c-0e01-4df1-b377-4c6bef856d64 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 28 00:42:40.288693 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:40.288693 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:40.288693 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:40.288693 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:40.288693 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:40.288693 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:40.288693 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:40.288693 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:40.288693 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:40.288693 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:40.288693 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:40.288693 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 28 00:42:40.288693 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 28 00:42:40.288693 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 28 00:42:40.288693 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 28 00:42:40.289879 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:40.289879 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:40.289879 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 28 00:42:40.289879 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 28 00:42:40.289879 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 28 00:42:40.289879 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 28 00:42:40.289879 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:40.289879 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:40.289879 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:40.289879 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:40.289879 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:40.289879 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:40.289879 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:40.289879 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:40.289879 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:40.289879 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:40.289879 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:40.289879 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:40.291040 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:40.291040 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:40.291040 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 28 00:42:40.291040 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 28 00:42:40.291040 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 28 00:42:40.291040 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 28 00:42:40.291040 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 28 00:42:40.291040 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 28 00:42:40.291040 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 28 00:42:40.291040 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:42:40.293608 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 464/927] 10.222.0.42 () {56 vars in 1117 bytes} [Sun Aug 28 00:42:40 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:40.299399 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3c40b1c-0e01-4df1-b377-4c6bef856d64 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:40.323479 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:40.342488 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eac9399-0e1e-4445-b379-0d73f8cdf46d None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:40.387852 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eff6b718-6224-446f-b62a-07b7dd06b947 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:40.388571 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eff6b718-6224-446f-b62a-07b7dd06b947 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:40.388889 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eff6b718-6224-446f-b62a-07b7dd06b947 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:40.389200 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eff6b718-6224-446f-b62a-07b7dd06b947 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:40.389794 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eff6b718-6224-446f-b62a-07b7dd06b947 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:40.391201 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eff6b718-6224-446f-b62a-07b7dd06b947 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:40.401109 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-eff6b718-6224-446f-b62a-07b7dd06b947 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 28 00:42:40.401109 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:40.401109 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 28 00:42:40.401109 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 28 00:42:40.401109 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 28 00:42:40.401109 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 28 00:42:40.401109 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 28 00:42:40.401109 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 28 00:42:40.401109 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 28 00:42:40.401109 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:40.401109 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 28 00:42:40.401109 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:40.401109 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:40.401109 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:40.401109 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:40.401109 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:40.401109 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:40.402169 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:40.402169 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:40.402169 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:40.402169 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:40.402169 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:40.402169 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:40.402169 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 28 00:42:40.402169 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 28 00:42:40.402169 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 28 00:42:40.402169 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 28 00:42:40.402169 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:40.402169 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:40.402169 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:40.402169 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:40.402169 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:40.402169 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:40.403028 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:40.403028 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:40.403028 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:40.403028 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:40.403028 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:40.403028 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:40.403028 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:40.403028 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:40.403028 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:40.403028 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:40.403028 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 28 00:42:40.403028 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 28 00:42:40.403028 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 28 00:42:40.403028 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:40.403028 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 464/928] 10.222.0.42 () {56 vars in 1126 bytes} [Sun Aug 28 00:42:40 2022] GET /identity/v3/services/object-store => generated 93 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:40.408258 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eff6b718-6224-446f-b62a-07b7dd06b947 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:40.450394 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3c40b1c-0e01-4df1-b377-4c6bef856d64 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:40.462147 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:40.496904 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-434455ca-146d-4572-9f02-c3773a425618 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:40.497608 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-434455ca-146d-4572-9f02-c3773a425618 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:40.497898 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-434455ca-146d-4572-9f02-c3773a425618 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:40.498728 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-434455ca-146d-4572-9f02-c3773a425618 None admin] PATH_INFO: `/v3/users` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:40.499832 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-434455ca-146d-4572-9f02-c3773a425618 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:40.500142 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-434455ca-146d-4572-9f02-c3773a425618 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:40.502655 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-434455ca-146d-4572-9f02-c3773a425618 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:40.527717 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 465/929] 10.222.0.42 () {56 vars in 1149 bytes} [Sun Aug 28 00:42:40 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 229 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 28 00:42:40.533492 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-434455ca-146d-4572-9f02-c3773a425618 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:40.541445 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eff6b718-6224-446f-b62a-07b7dd06b947 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:40.579093 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfc81986-4dbf-422e-aa48-28af6e3d2145 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:40.580039 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfc81986-4dbf-422e-aa48-28af6e3d2145 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:40.580367 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfc81986-4dbf-422e-aa48-28af6e3d2145 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:40.580642 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfc81986-4dbf-422e-aa48-28af6e3d2145 None admin] PATH_INFO: `/v3/services` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:40.581318 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfc81986-4dbf-422e-aa48-28af6e3d2145 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:40.581621 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfc81986-4dbf-422e-aa48-28af6e3d2145 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:40.583003 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfc81986-4dbf-422e-aa48-28af6e3d2145 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:40.588685 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 465/930] 10.222.0.42 () {56 vars in 1122 bytes} [Sun Aug 28 00:42:40 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 28 00:42:40.594627 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfc81986-4dbf-422e-aa48-28af6e3d2145 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:40.609627 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:40.641918 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-434455ca-146d-4572-9f02-c3773a425618 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:40.677371 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-333997a4-d7bb-4c42-9b9b-78ff2fdae821 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:40.678013 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-333997a4-d7bb-4c42-9b9b-78ff2fdae821 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:40.678292 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-333997a4-d7bb-4c42-9b9b-78ff2fdae821 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:40.678562 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-333997a4-d7bb-4c42-9b9b-78ff2fdae821 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:40.689574 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-333997a4-d7bb-4c42-9b9b-78ff2fdae821 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:40.691122 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-333997a4-d7bb-4c42-9b9b-78ff2fdae821 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:40.703267 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfc81986-4dbf-422e-aa48-28af6e3d2145 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:40.705115 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:40.707648 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-333997a4-d7bb-4c42-9b9b-78ff2fdae821 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:42:40.707648 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:40.707648 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:40.707648 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:40.707648 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:40.707648 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:40.707648 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:40.707648 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:40.707648 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:40.707648 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:40.707648 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:40.707648 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:40.707648 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 28 00:42:40.707648 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 28 00:42:40.707648 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 28 00:42:40.707648 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 28 00:42:40.708802 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:40.708802 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:40.708802 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:40.708802 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:40.708802 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:40.708802 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:40.708802 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:40.708802 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:40.708802 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:40.708802 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:40.708802 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:40.708802 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:40.708802 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:40.708802 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:40.708802 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:40.708802 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:40.708802 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 28 00:42:40.709859 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 28 00:42:40.709859 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:42:40.709859 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:42:40.714372 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 466/931] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:42:40 2022] GET /identity/v3/domains/Default => generated 87 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 28 00:42:40.715989 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-333997a4-d7bb-4c42-9b9b-78ff2fdae821 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:40.745111 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2db5e885-129e-49af-8199-c65360f4ce9d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:40.746038 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2db5e885-129e-49af-8199-c65360f4ce9d None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:40.746462 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2db5e885-129e-49af-8199-c65360f4ce9d None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:40.746873 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2db5e885-129e-49af-8199-c65360f4ce9d None admin] PATH_INFO: `/v3/services` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:40.747687 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2db5e885-129e-49af-8199-c65360f4ce9d None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:40.748253 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2db5e885-129e-49af-8199-c65360f4ce9d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:40.749961 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2db5e885-129e-49af-8199-c65360f4ce9d None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:40.760673 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 466/932] 10.222.0.42 () {56 vars in 1122 bytes} [Sun Aug 28 00:42:40 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:40.761879 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2db5e885-129e-49af-8199-c65360f4ce9d None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:40.828103 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-333997a4-d7bb-4c42-9b9b-78ff2fdae821 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:40.856240 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:40.870053 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8580f41f-13b7-47c9-aff2-316e8396b63e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:40.870665 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8580f41f-13b7-47c9-aff2-316e8396b63e None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:40.870942 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8580f41f-13b7-47c9-aff2-316e8396b63e None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:40.871211 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8580f41f-13b7-47c9-aff2-316e8396b63e None admin] PATH_INFO: `/v3/domains` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:40.871860 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8580f41f-13b7-47c9-aff2-316e8396b63e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:40.872170 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8580f41f-13b7-47c9-aff2-316e8396b63e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:40.873668 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8580f41f-13b7-47c9-aff2-316e8396b63e None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:40.891919 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 467/933] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Aug 28 00:42:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:40.892951 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8580f41f-13b7-47c9-aff2-316e8396b63e None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:40.911668 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2db5e885-129e-49af-8199-c65360f4ce9d None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:40.967785 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9a8224c-04ea-47ad-87bd-728cbcd6beb7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:40.969430 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9a8224c-04ea-47ad-87bd-728cbcd6beb7 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:40.969712 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9a8224c-04ea-47ad-87bd-728cbcd6beb7 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:40.969980 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9a8224c-04ea-47ad-87bd-728cbcd6beb7 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:40.970746 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9a8224c-04ea-47ad-87bd-728cbcd6beb7 None admin] RBAC: Adding query filter params (interface=public, service_id=2ba2936a2af44b56a45e060abc738be4, region_id=RegionOne) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:40.971020 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9a8224c-04ea-47ad-87bd-728cbcd6beb7 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:40.972420 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9a8224c-04ea-47ad-87bd-728cbcd6beb7 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:40.978366 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 467/934] 10.222.0.42 () {56 vars in 1251 bytes} [Sun Aug 28 00:42:40 2022] GET /identity/v3/endpoints?service_id=2ba2936a2af44b56a45e060abc738be4&interface=public®ion_id=RegionOne => generated 194 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:40.979732 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac2bb82a-96d2-45b2-906c-757ea73b78b4 None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:40.980016 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac2bb82a-96d2-45b2-906c-757ea73b78b4 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:40.980298 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac2bb82a-96d2-45b2-906c-757ea73b78b4 None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:40.981810 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 468/935] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:42:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:42:40.983930 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac2bb82a-96d2-45b2-906c-757ea73b78b4 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:41.009584 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8580f41f-13b7-47c9-aff2-316e8396b63e None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:41.017688 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:41.050363 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-425d6e05-7db0-4d51-a5c6-498324ba599d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:41.051090 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-425d6e05-7db0-4d51-a5c6-498324ba599d None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:41.051392 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-425d6e05-7db0-4d51-a5c6-498324ba599d None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:41.051631 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-425d6e05-7db0-4d51-a5c6-498324ba599d None admin] PATH_INFO: `/v3/projects/service` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:41.058676 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-425d6e05-7db0-4d51-a5c6-498324ba599d None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:41.060148 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-425d6e05-7db0-4d51-a5c6-498324ba599d None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:41.072070 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-425d6e05-7db0-4d51-a5c6-498324ba599d None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 28 00:42:41.072070 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:41.072070 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:41.072070 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:41.072070 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:41.072070 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:41.072070 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:41.072070 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:41.072070 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:41.072070 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:41.072070 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:41.072070 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:41.072070 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 28 00:42:41.072070 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 28 00:42:41.072070 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 28 00:42:41.073839 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 28 00:42:41.073839 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:41.073839 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:41.073839 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:41.073839 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:41.073839 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:41.073839 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:41.073839 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:41.073839 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:41.073839 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:41.073839 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:41.073839 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:41.073839 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:41.073839 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:41.073839 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:41.073839 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:41.073839 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:41.075601 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 28 00:42:41.075601 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 28 00:42:41.075601 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 28 00:42:41.075601 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 28 00:42:41.075601 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 28 00:42:41.075601 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 28 00:42:41.075601 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 28 00:42:41.075601 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:42:41.075601 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 468/936] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Aug 28 00:42:40 2022] GET /identity/v3/projects/service => generated 88 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:41.075601 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e4680e9-22c1-40f3-9216-2e51ca44ce1e None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:41.075601 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e4680e9-22c1-40f3-9216-2e51ca44ce1e None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:41.076787 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e4680e9-22c1-40f3-9216-2e51ca44ce1e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:41.095185 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-2e4680e9-22c1-40f3-9216-2e51ca44ce1e None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:42:41.096567 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-2e4680e9-22c1-40f3-9216-2e51ca44ce1e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:42:41.113626 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac2bb82a-96d2-45b2-906c-757ea73b78b4 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:41.150952 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:41.165751 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90b764e2-c207-4478-a4c4-ea4ee0d22719 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:41.166924 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90b764e2-c207-4478-a4c4-ea4ee0d22719 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:41.167416 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90b764e2-c207-4478-a4c4-ea4ee0d22719 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:41.167883 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90b764e2-c207-4478-a4c4-ea4ee0d22719 None admin] PATH_INFO: `/v3/services` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:41.168998 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90b764e2-c207-4478-a4c4-ea4ee0d22719 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:41.171655 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90b764e2-c207-4478-a4c4-ea4ee0d22719 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:41.182981 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 469/937] 10.222.0.42 () {56 vars in 1087 bytes} [Sun Aug 28 00:42:40 2022] GET /identity/v3/services => generated 1497 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 00:42:41.184444 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90b764e2-c207-4478-a4c4-ea4ee0d22719 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:41.208685 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 469/938] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:42:41 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 28 00:42:41.316840 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:41.323918 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90b764e2-c207-4478-a4c4-ea4ee0d22719 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:41.367272 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8a6a288-7504-47e5-a56b-793201cc1bb0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:41.367935 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8a6a288-7504-47e5-a56b-793201cc1bb0 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:41.368166 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8a6a288-7504-47e5-a56b-793201cc1bb0 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:41.368440 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8a6a288-7504-47e5-a56b-793201cc1bb0 None admin] PATH_INFO: `/v3/projects` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:41.369290 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8a6a288-7504-47e5-a56b-793201cc1bb0 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:41.369632 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8a6a288-7504-47e5-a56b-793201cc1bb0 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:41.371217 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8a6a288-7504-47e5-a56b-793201cc1bb0 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:41.385275 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 470/939] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Aug 28 00:42:41 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:41.390996 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e4680e9-22c1-40f3-9216-2e51ca44ce1e None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:41.496229 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8a6a288-7504-47e5-a56b-793201cc1bb0 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:41.569639 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e4680e9-22c1-40f3-9216-2e51ca44ce1e None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:41.611866 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8a6a288-7504-47e5-a56b-793201cc1bb0 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:41.617521 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca78f251-1784-4be7-9bc5-8834bfc77a6f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:41.618452 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca78f251-1784-4be7-9bc5-8834bfc77a6f None admin] REQUEST_METHOD: `PUT` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:41.618922 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca78f251-1784-4be7-9bc5-8834bfc77a6f None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:41.619343 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca78f251-1784-4be7-9bc5-8834bfc77a6f None admin] PATH_INFO: `/v3/projects/f53e6fdb369b480fbebb4086a505df67/users/eede08f020e04282bb994b855e371433/roles/2c6107d249594bd3abf791034b71e98f` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:41.622400 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca78f251-1784-4be7-9bc5-8834bfc77a6f None admin] RBAC: Authorizing `identity:create_grant(project_id=f53e6fdb369b480fbebb4086a505df67, user_id=eede08f020e04282bb994b855e371433, role_id=2c6107d249594bd3abf791034b71e98f)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:41.624599 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca78f251-1784-4be7-9bc5-8834bfc77a6f None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:41.645790 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 470/940] 10.222.0.42 () {58 vars in 1439 bytes} [Sun Aug 28 00:42:41 2022] PUT /identity/v3/projects/f53e6fdb369b480fbebb4086a505df67/users/eede08f020e04282bb994b855e371433/roles/2c6107d249594bd3abf791034b71e98f => generated 0 bytes in 255 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 28 00:42:41.660612 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-422a85e9-fabe-4dbf-a3d1-a621178b83b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:41.660612 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-422a85e9-fabe-4dbf-a3d1-a621178b83b0 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:41.660612 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-422a85e9-fabe-4dbf-a3d1-a621178b83b0 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:41.660612 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-422a85e9-fabe-4dbf-a3d1-a621178b83b0 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:41.661245 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-422a85e9-fabe-4dbf-a3d1-a621178b83b0 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:41.661918 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-422a85e9-fabe-4dbf-a3d1-a621178b83b0 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:41.673293 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-422a85e9-fabe-4dbf-a3d1-a621178b83b0 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 28 00:42:41.673293 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:41.673293 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 28 00:42:41.673293 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 28 00:42:41.673293 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 28 00:42:41.673293 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 28 00:42:41.673293 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 28 00:42:41.673293 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 28 00:42:41.673293 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 28 00:42:41.673293 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:41.673293 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 28 00:42:41.673293 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:41.673293 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:41.673293 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:41.673293 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:41.673293 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:41.674383 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:41.674383 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:41.674383 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:41.674383 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:41.674383 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:41.674383 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:41.674383 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:41.674383 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 28 00:42:41.674383 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 28 00:42:41.674383 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 28 00:42:41.674383 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 28 00:42:41.674383 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:41.674383 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:41.674383 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:41.674383 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:41.674383 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:41.674383 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:41.675287 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:41.675287 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:41.675287 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:41.675287 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:41.675287 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:41.675287 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:41.675287 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:41.675287 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:41.675287 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:41.675287 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:41.675287 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 28 00:42:41.675287 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 28 00:42:41.675287 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Aug 28 00:42:41.675287 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:41.675287 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 471/941] 10.222.0.42 () {56 vars in 1132 bytes} [Sun Aug 28 00:42:41 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 28 00:42:41.682059 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca78f251-1784-4be7-9bc5-8834bfc77a6f None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:41.731830 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:41.817677 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca78f251-1784-4be7-9bc5-8834bfc77a6f None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:41.853959 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41b5f41a-3699-4c88-975f-9b7ec9c9c57a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:41.854493 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b5f41a-3699-4c88-975f-9b7ec9c9c57a None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:41.854770 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b5f41a-3699-4c88-975f-9b7ec9c9c57a None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:41.855037 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b5f41a-3699-4c88-975f-9b7ec9c9c57a None admin] PATH_INFO: `/v3/services` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:41.855661 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41b5f41a-3699-4c88-975f-9b7ec9c9c57a None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:41.855957 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41b5f41a-3699-4c88-975f-9b7ec9c9c57a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:41.857490 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41b5f41a-3699-4c88-975f-9b7ec9c9c57a None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:41.862865 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 471/942] 10.222.0.42 () {56 vars in 1126 bytes} [Sun Aug 28 00:42:41 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:41.868370 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-422a85e9-fabe-4dbf-a3d1-a621178b83b0 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:41.891467 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:41.983801 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-422a85e9-fabe-4dbf-a3d1-a621178b83b0 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:42.022997 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74a891d6-46ac-41c1-8c19-e843898ef466 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:42.023912 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74a891d6-46ac-41c1-8c19-e843898ef466 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:42.024370 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74a891d6-46ac-41c1-8c19-e843898ef466 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:42.024856 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74a891d6-46ac-41c1-8c19-e843898ef466 None admin] PATH_INFO: `/v3/services` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:42.025672 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74a891d6-46ac-41c1-8c19-e843898ef466 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:42.026188 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74a891d6-46ac-41c1-8c19-e843898ef466 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:42.027725 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74a891d6-46ac-41c1-8c19-e843898ef466 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:42.033672 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 472/943] 10.222.0.42 () {56 vars in 1126 bytes} [Sun Aug 28 00:42:41 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:42.041013 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41b5f41a-3699-4c88-975f-9b7ec9c9c57a None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:42.151854 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:42.156855 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41b5f41a-3699-4c88-975f-9b7ec9c9c57a None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:42.198359 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5c5d15a-1aba-4c31-9e44-2fa8a9bd9ed2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:42.199035 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5c5d15a-1aba-4c31-9e44-2fa8a9bd9ed2 None admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:42.199318 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5c5d15a-1aba-4c31-9e44-2fa8a9bd9ed2 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:42.199589 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5c5d15a-1aba-4c31-9e44-2fa8a9bd9ed2 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:42.200305 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5c5d15a-1aba-4c31-9e44-2fa8a9bd9ed2 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:42.201921 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5c5d15a-1aba-4c31-9e44-2fa8a9bd9ed2 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:42.226035 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 472/944] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 00:42:42 2022] POST /identity/v3/endpoints => generated 347 bytes in 187 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:42:42.335892 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-059ed038-c761-42be-9d42-3e74f05e6d2f None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:42.336255 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-059ed038-c761-42be-9d42-3e74f05e6d2f None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:42.336497 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-059ed038-c761-42be-9d42-3e74f05e6d2f None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:42.338229 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 473/945] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:42:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:42:42.344676 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72ed3a8d-a7a4-4c7c-bafc-f82c30c8626b None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:42.345690 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72ed3a8d-a7a4-4c7c-bafc-f82c30c8626b None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:42.346281 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72ed3a8d-a7a4-4c7c-bafc-f82c30c8626b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:42.370046 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-72ed3a8d-a7a4-4c7c-bafc-f82c30c8626b None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:42:42.371575 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-72ed3a8d-a7a4-4c7c-bafc-f82c30c8626b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:42:42.463971 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 473/946] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:42:42 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 28 00:42:42.780742 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-059ed038-c761-42be-9d42-3e74f05e6d2f None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:42.878715 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:42.899065 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-059ed038-c761-42be-9d42-3e74f05e6d2f None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:42.935754 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cf625eb-f25b-47f0-b871-6e519196b28a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:42.936388 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cf625eb-f25b-47f0-b871-6e519196b28a None admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:42.936669 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cf625eb-f25b-47f0-b871-6e519196b28a None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:42.936937 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cf625eb-f25b-47f0-b871-6e519196b28a None admin] PATH_INFO: `/v3/projects` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:42.937626 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cf625eb-f25b-47f0-b871-6e519196b28a None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:42.939159 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cf625eb-f25b-47f0-b871-6e519196b28a None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:42.941914 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: WARNING keystone.server.flask.common [None req-9cf625eb-f25b-47f0-b871-6e519196b28a None admin] Deprecated: Not specifying a domain during a create user, group or project call, and relying on falling back to the default domain, is deprecated as of Liberty. There is no plan to remove this compatibility, however, future API versions may remove this, so please specify the domain explicitly or use a domain-scoped token. Aug 28 00:42:42.961916 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.sql.core [None req-9cf625eb-f25b-47f0-b871-6e519196b28a None admin] Conflict project: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Aug 28 00:42:42.961916 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Aug 28 00:42:42.961916 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [parameters: {'id': '6d9eff524a2d47cab4e0fa04a545fcbd', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Aug 28 00:42:42.961916 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81507) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Aug 28 00:42:42.970013 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-9cf625eb-f25b-47f0-b871-6e519196b28a None admin] Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 6d9eff524a2d47cab4e0fa04a545fcbd.: keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 6d9eff524a2d47cab4e0fa04a545fcbd. Aug 28 00:42:42.970013 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:42.970013 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Aug 28 00:42:42.970013 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application self.dialect.do_execute( Aug 28 00:42:42.970013 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Aug 28 00:42:42.970013 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Aug 28 00:42:42.970013 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Aug 28 00:42:42.970013 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application result = self._query(query) Aug 28 00:42:42.970013 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Aug 28 00:42:42.970013 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application conn.query(q) Aug 28 00:42:42.970013 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Aug 28 00:42:42.970013 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Aug 28 00:42:42.970013 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Aug 28 00:42:42.971262 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application result.read() Aug 28 00:42:42.971262 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Aug 28 00:42:42.971262 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Aug 28 00:42:42.971262 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Aug 28 00:42:42.971262 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application packet.raise_for_error() Aug 28 00:42:42.971262 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Aug 28 00:42:42.971262 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Aug 28 00:42:42.971262 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Aug 28 00:42:42.971262 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Aug 28 00:42:42.971262 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Aug 28 00:42:42.971262 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:42.971262 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Aug 28 00:42:42.971262 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:42.971262 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:42.971262 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Aug 28 00:42:42.971262 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return method(*args, **kwargs) Aug 28 00:42:42.971262 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 241, in create_project Aug 28 00:42:42.971262 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return project_ref.to_dict() Aug 28 00:42:42.972315 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Aug 28 00:42:42.972315 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application next(self.gen) Aug 28 00:42:42.972315 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Aug 28 00:42:42.972315 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application yield resource Aug 28 00:42:42.972315 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Aug 28 00:42:42.972315 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application next(self.gen) Aug 28 00:42:42.972315 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Aug 28 00:42:42.972315 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application self.session.rollback() Aug 28 00:42:42.972315 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Aug 28 00:42:42.972315 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application self.force_reraise() Aug 28 00:42:42.972315 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Aug 28 00:42:42.972315 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise self.value Aug 28 00:42:42.972315 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Aug 28 00:42:42.972315 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Aug 28 00:42:42.972315 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Aug 28 00:42:42.972315 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application session.commit() Aug 28 00:42:42.972315 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Aug 28 00:42:42.973301 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Aug 28 00:42:42.973301 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Aug 28 00:42:42.973301 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application self._prepare_impl() Aug 28 00:42:42.973301 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Aug 28 00:42:42.973301 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application self.session.flush() Aug 28 00:42:42.973301 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Aug 28 00:42:42.973301 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application self._flush(objects) Aug 28 00:42:42.973301 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Aug 28 00:42:42.973301 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Aug 28 00:42:42.973301 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Aug 28 00:42:42.973301 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application compat.raise_( Aug 28 00:42:42.973301 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Aug 28 00:42:42.973301 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception Aug 28 00:42:42.973301 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Aug 28 00:42:42.973301 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application flush_context.execute() Aug 28 00:42:42.973301 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Aug 28 00:42:42.973301 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rec.execute(self) Aug 28 00:42:42.974215 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Aug 28 00:42:42.974215 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Aug 28 00:42:42.974215 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Aug 28 00:42:42.974215 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application _emit_insert_statements( Aug 28 00:42:42.974215 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Aug 28 00:42:42.974215 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application c = connection._execute_20( Aug 28 00:42:42.974215 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Aug 28 00:42:42.974215 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Aug 28 00:42:42.974215 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Aug 28 00:42:42.974215 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Aug 28 00:42:42.974215 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Aug 28 00:42:42.974215 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ret = self._execute_context( Aug 28 00:42:42.974215 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Aug 28 00:42:42.974215 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Aug 28 00:42:42.974215 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Aug 28 00:42:42.976473 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Aug 28 00:42:42.976473 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Aug 28 00:42:42.976473 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception Aug 28 00:42:42.976473 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Aug 28 00:42:42.976473 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application self.dialect.do_execute( Aug 28 00:42:42.976473 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Aug 28 00:42:42.976473 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Aug 28 00:42:42.976473 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Aug 28 00:42:42.976473 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application result = self._query(query) Aug 28 00:42:42.976473 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Aug 28 00:42:42.976473 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application conn.query(q) Aug 28 00:42:42.976473 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Aug 28 00:42:42.976473 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Aug 28 00:42:42.976473 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Aug 28 00:42:42.976473 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application result.read() Aug 28 00:42:42.976473 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Aug 28 00:42:42.976473 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Aug 28 00:42:42.978858 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Aug 28 00:42:42.978858 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application packet.raise_for_error() Aug 28 00:42:42.978858 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Aug 28 00:42:42.978858 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Aug 28 00:42:42.978858 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Aug 28 00:42:42.978858 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Aug 28 00:42:42.978858 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Aug 28 00:42:42.978858 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Aug 28 00:42:42.978858 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [parameters: {'id': '6d9eff524a2d47cab4e0fa04a545fcbd', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Aug 28 00:42:42.978858 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Aug 28 00:42:42.978858 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:42.978858 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 28 00:42:42.978858 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:42.978858 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:42.978858 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 216, in create_project Aug 28 00:42:42.982077 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ret = self.driver.create_project(project_id, project) Aug 28 00:42:42.982077 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Aug 28 00:42:42.982077 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Aug 28 00:42:42.982077 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - Duplicate entry found with name service at domain ID default. Aug 28 00:42:42.982077 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:42.982077 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 28 00:42:42.982077 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:42.982077 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:42.982077 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:42.982077 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:42.982077 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:42.982077 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:42.982077 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:42.982077 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:42.982077 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:42.982077 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:42.982077 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:42.985109 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:42.985109 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 176, in post Aug 28 00:42:42.985109 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ref = PROVIDERS.resource_api.create_project( Aug 28 00:42:42.985109 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:42.985109 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:42.985109 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 218, in create_project Aug 28 00:42:42.985109 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.Conflict( Aug 28 00:42:42.985109 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 6d9eff524a2d47cab4e0fa04a545fcbd. Aug 28 00:42:42.985109 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:42.985109 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 474/947] 10.222.0.42 () {60 vars in 1140 bytes} [Sun Aug 28 00:42:42 2022] POST /identity/v3/projects => generated 262 bytes in 191 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:42.986498 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72ed3a8d-a7a4-4c7c-bafc-f82c30c8626b None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:43.067885 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:43.090020 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72ed3a8d-a7a4-4c7c-bafc-f82c30c8626b None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:43.144793 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-678b2754-ecdb-4082-868d-10967ff41302 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:43.145833 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-678b2754-ecdb-4082-868d-10967ff41302 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:43.146291 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-678b2754-ecdb-4082-868d-10967ff41302 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:43.146727 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-678b2754-ecdb-4082-868d-10967ff41302 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:43.154999 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-678b2754-ecdb-4082-868d-10967ff41302 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:43.156815 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-678b2754-ecdb-4082-868d-10967ff41302 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:43.169858 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-678b2754-ecdb-4082-868d-10967ff41302 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 28 00:42:43.169858 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:43.169858 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:43.169858 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:43.169858 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:43.169858 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:43.169858 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:43.169858 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:43.169858 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:43.169858 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:43.169858 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:43.169858 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:43.169858 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 28 00:42:43.169858 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 28 00:42:43.169858 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 28 00:42:43.171026 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 28 00:42:43.171026 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:43.171026 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:43.171026 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:43.171026 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:43.171026 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:43.171026 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:43.171026 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:43.171026 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:43.171026 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:43.171026 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:43.171026 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:43.171026 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:43.171026 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:43.171026 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:43.171026 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:43.171026 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:43.172249 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 28 00:42:43.172249 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 28 00:42:43.172249 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 28 00:42:43.172249 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 28 00:42:43.172249 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 28 00:42:43.172249 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 28 00:42:43.172249 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 28 00:42:43.172249 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:42:43.174382 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 474/948] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Aug 28 00:42:42 2022] GET /identity/v3/projects/service => generated 88 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:43.182279 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cf625eb-f25b-47f0-b871-6e519196b28a None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:43.244738 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acae3e0a-5f72-4539-aa0e-28838dd009d5 None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:43.245324 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acae3e0a-5f72-4539-aa0e-28838dd009d5 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:43.245599 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acae3e0a-5f72-4539-aa0e-28838dd009d5 None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:43.247176 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 475/949] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:42:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:42:43.252825 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-245fff3c-9c16-4a7f-bd99-4f59ce978129 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:43.253426 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-245fff3c-9c16-4a7f-bd99-4f59ce978129 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:43.253922 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-245fff3c-9c16-4a7f-bd99-4f59ce978129 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:43.273963 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-245fff3c-9c16-4a7f-bd99-4f59ce978129 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:42:43.275594 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-245fff3c-9c16-4a7f-bd99-4f59ce978129 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:42:43.305093 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cf625eb-f25b-47f0-b871-6e519196b28a None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:43.342875 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fed6e2a-6785-4e75-83bb-b6f48c3b381e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:43.343415 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fed6e2a-6785-4e75-83bb-b6f48c3b381e None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:43.343745 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fed6e2a-6785-4e75-83bb-b6f48c3b381e None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:43.344025 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fed6e2a-6785-4e75-83bb-b6f48c3b381e None admin] PATH_INFO: `/v3/projects` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:43.344700 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fed6e2a-6785-4e75-83bb-b6f48c3b381e None admin] RBAC: Adding query filter params (name=service) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:43.345027 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fed6e2a-6785-4e75-83bb-b6f48c3b381e None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:43.346749 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fed6e2a-6785-4e75-83bb-b6f48c3b381e None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:43.358381 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:43.361632 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 476/950] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:42:43 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 28 00:42:43.373010 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 475/951] 10.222.0.42 () {56 vars in 1112 bytes} [Sun Aug 28 00:42:43 2022] GET /identity/v3/projects?name=service => generated 411 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:43.626378 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-245fff3c-9c16-4a7f-bd99-4f59ce978129 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:43.705300 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:43.739144 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-245fff3c-9c16-4a7f-bd99-4f59ce978129 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:43.775512 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4629271-0928-4867-bcac-216a2638ae4b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:43.776555 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4629271-0928-4867-bcac-216a2638ae4b None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:43.777087 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4629271-0928-4867-bcac-216a2638ae4b None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:43.777563 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4629271-0928-4867-bcac-216a2638ae4b None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:43.778368 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4629271-0928-4867-bcac-216a2638ae4b None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:43.780015 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4629271-0928-4867-bcac-216a2638ae4b None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:43.785330 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-824f84c7-7cf5-4e9d-9381-008980a368e0 None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:43.785839 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-824f84c7-7cf5-4e9d-9381-008980a368e0 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:43.785927 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-824f84c7-7cf5-4e9d-9381-008980a368e0 None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:43.787494 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 476/952] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:42:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 28 00:42:43.791193 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-d4629271-0928-4867-bcac-216a2638ae4b None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 28 00:42:43.791193 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:43.791193 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 28 00:42:43.791193 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 28 00:42:43.791193 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 28 00:42:43.791193 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 28 00:42:43.791193 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 28 00:42:43.791193 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 28 00:42:43.791193 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 28 00:42:43.791193 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:42:43.791193 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 28 00:42:43.791193 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:42:43.791193 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:43.791193 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:43.791193 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:43.791193 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:43.791193 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:43.792410 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:43.792410 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:43.792410 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:43.792410 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:43.792410 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:43.792410 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:43.792410 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 28 00:42:43.792410 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 28 00:42:43.792410 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 28 00:42:43.792410 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 28 00:42:43.792410 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:43.792410 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:43.792410 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:43.792410 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:43.792410 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:43.792410 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:43.793374 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:43.793374 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:43.793374 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:43.793374 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:43.793374 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:43.793374 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:43.793374 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:43.793374 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:43.793374 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:43.793374 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:43.793374 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 28 00:42:43.793374 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 28 00:42:43.793374 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 28 00:42:43.793374 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:42:43.794170 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a28274c-5dbe-47a6-ad0f-eb77060bb49c None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:43.794170 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a28274c-5dbe-47a6-ad0f-eb77060bb49c None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:43.794170 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a28274c-5dbe-47a6-ad0f-eb77060bb49c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:43.796577 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 477/953] 10.222.0.42 () {56 vars in 1126 bytes} [Sun Aug 28 00:42:43 2022] GET /identity/v3/services/object-store => generated 93 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:43.802242 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4629271-0928-4867-bcac-216a2638ae4b None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:43.818373 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-1a28274c-5dbe-47a6-ad0f-eb77060bb49c None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:42:43.821233 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-1a28274c-5dbe-47a6-ad0f-eb77060bb49c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:42:43.917597 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 477/954] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:42:43 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 28 00:42:43.926668 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4629271-0928-4867-bcac-216a2638ae4b None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:43.964612 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-057483d1-b875-4784-b76d-9705e7c09a84 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:43.965544 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-057483d1-b875-4784-b76d-9705e7c09a84 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:43.965997 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-057483d1-b875-4784-b76d-9705e7c09a84 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:43.966427 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-057483d1-b875-4784-b76d-9705e7c09a84 None admin] PATH_INFO: `/v3/services` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:43.967317 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-057483d1-b875-4784-b76d-9705e7c09a84 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:43.967769 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-057483d1-b875-4784-b76d-9705e7c09a84 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:43.969351 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-057483d1-b875-4784-b76d-9705e7c09a84 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:43.970752 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:43.975866 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 478/955] 10.222.0.42 () {56 vars in 1122 bytes} [Sun Aug 28 00:42:43 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 28 00:42:43.981523 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a28274c-5dbe-47a6-ad0f-eb77060bb49c None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:44.084658 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a28274c-5dbe-47a6-ad0f-eb77060bb49c None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:44.125361 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:44.133100 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-720c2b80-23be-4c52-a76f-8b3c27754c9e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:44.133771 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-720c2b80-23be-4c52-a76f-8b3c27754c9e None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:44.134059 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-720c2b80-23be-4c52-a76f-8b3c27754c9e None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:44.134340 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-720c2b80-23be-4c52-a76f-8b3c27754c9e None admin] PATH_INFO: `/v3/services` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:44.134992 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-720c2b80-23be-4c52-a76f-8b3c27754c9e None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:44.135299 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-720c2b80-23be-4c52-a76f-8b3c27754c9e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:44.136746 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-720c2b80-23be-4c52-a76f-8b3c27754c9e None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:44.142674 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 478/956] 10.222.0.42 () {56 vars in 1122 bytes} [Sun Aug 28 00:42:43 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:44.148064 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-057483d1-b875-4784-b76d-9705e7c09a84 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:44.195621 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-720c2b80-23be-4c52-a76f-8b3c27754c9e None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:44.270655 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-057483d1-b875-4784-b76d-9705e7c09a84 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:44.306570 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-720c2b80-23be-4c52-a76f-8b3c27754c9e None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:44.312978 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fb9a87c-40b7-4163-bddc-21172c14a179 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:44.313601 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb9a87c-40b7-4163-bddc-21172c14a179 None admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:44.314313 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb9a87c-40b7-4163-bddc-21172c14a179 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:44.314587 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb9a87c-40b7-4163-bddc-21172c14a179 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:44.315258 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fb9a87c-40b7-4163-bddc-21172c14a179 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:44.317335 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fb9a87c-40b7-4163-bddc-21172c14a179 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:44.341113 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 479/957] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 00:42:44 2022] POST /identity/v3/endpoints => generated 349 bytes in 194 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:42:44.342726 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fdd1897-e4dd-4b71-bbe4-2f9031e3056f None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:44.343250 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fdd1897-e4dd-4b71-bbe4-2f9031e3056f None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:44.343751 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fdd1897-e4dd-4b71-bbe4-2f9031e3056f None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:44.345614 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 480/958] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:42:44 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:42:44.353586 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62731a3a-92b1-41cc-9321-26e5eb451fbf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:44.353586 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62731a3a-92b1-41cc-9321-26e5eb451fbf None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:44.353586 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62731a3a-92b1-41cc-9321-26e5eb451fbf None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:44.353586 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62731a3a-92b1-41cc-9321-26e5eb451fbf None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:44.355950 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f85f5868-a4ad-4634-837c-7837b84e03a7 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:44.356588 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f85f5868-a4ad-4634-837c-7837b84e03a7 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:44.357119 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f85f5868-a4ad-4634-837c-7837b84e03a7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:44.366373 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62731a3a-92b1-41cc-9321-26e5eb451fbf None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 28 00:42:44.371504 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-62731a3a-92b1-41cc-9321-26e5eb451fbf None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 28 00:42:44.371504 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:44.371504 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:44.371504 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:44.371504 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:44.371504 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:44.371504 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:44.371504 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:44.371504 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:44.371504 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:44.371504 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:44.371504 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:44.371504 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 28 00:42:44.371504 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 28 00:42:44.371504 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 28 00:42:44.371504 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 28 00:42:44.372689 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 28 00:42:44.372689 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 28 00:42:44.372689 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 28 00:42:44.372689 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 28 00:42:44.372689 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:44.372689 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:44.372689 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:44.372689 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:44.372689 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:44.372689 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:44.372689 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:44.372689 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:44.372689 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:44.372689 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:44.372689 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:44.372689 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:44.378183 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:44.378183 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:44.378183 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:44.378183 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:44.378183 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 28 00:42:44.378183 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 28 00:42:44.378183 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 28 00:42:44.378183 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 28 00:42:44.378183 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 28 00:42:44.378183 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 28 00:42:44.378183 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 28 00:42:44.378183 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:44.378183 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 479/959] 10.222.0.42 () {56 vars in 1120 bytes} [Sun Aug 28 00:42:44 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 28 00:42:44.380056 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62731a3a-92b1-41cc-9321-26e5eb451fbf None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:44.394588 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-f85f5868-a4ad-4634-837c-7837b84e03a7 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:42:44.396373 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-f85f5868-a4ad-4634-837c-7837b84e03a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:42:44.439274 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:44.461564 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:44.487608 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 481/960] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:42:44 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 28 00:42:44.506315 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62731a3a-92b1-41cc-9321-26e5eb451fbf None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:44.545752 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f88e2dd-a3cd-4f09-ac3f-8eebf84dd821 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:44.546682 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f88e2dd-a3cd-4f09-ac3f-8eebf84dd821 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:44.547141 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f88e2dd-a3cd-4f09-ac3f-8eebf84dd821 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:44.547617 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f88e2dd-a3cd-4f09-ac3f-8eebf84dd821 None admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:44.548467 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f88e2dd-a3cd-4f09-ac3f-8eebf84dd821 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:44.549000 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f88e2dd-a3cd-4f09-ac3f-8eebf84dd821 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:44.550589 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f88e2dd-a3cd-4f09-ac3f-8eebf84dd821 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:44.559754 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 480/961] 10.222.0.42 () {56 vars in 1115 bytes} [Sun Aug 28 00:42:44 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:44.565223 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f85f5868-a4ad-4634-837c-7837b84e03a7 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:44.677150 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f85f5868-a4ad-4634-837c-7837b84e03a7 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:44.718377 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2393ddc4-502e-4ecb-8abb-1d694d0c7a24 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:44.718942 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2393ddc4-502e-4ecb-8abb-1d694d0c7a24 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:44.719222 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2393ddc4-502e-4ecb-8abb-1d694d0c7a24 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:44.719492 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2393ddc4-502e-4ecb-8abb-1d694d0c7a24 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:44.730553 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2393ddc4-502e-4ecb-8abb-1d694d0c7a24 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:44.732030 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2393ddc4-502e-4ecb-8abb-1d694d0c7a24 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:44.748661 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-2393ddc4-502e-4ecb-8abb-1d694d0c7a24 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:42:44.748661 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:44.748661 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:44.748661 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:44.748661 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:44.748661 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:44.748661 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:44.748661 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:44.748661 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:44.748661 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:44.748661 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:44.748661 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:44.748661 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 28 00:42:44.748661 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 28 00:42:44.748661 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 28 00:42:44.748661 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 28 00:42:44.749769 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:44.749769 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:44.749769 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:44.749769 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:44.749769 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:44.749769 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:44.749769 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:44.749769 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:44.749769 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:44.749769 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:44.749769 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:44.749769 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:44.749769 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:44.749769 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:44.749769 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:44.749769 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:44.749769 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 28 00:42:44.750679 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 28 00:42:44.750679 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:42:44.750679 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:42:44.750679 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 482/962] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:42:44 2022] GET /identity/v3/domains/Default => generated 87 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 28 00:42:44.756095 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f88e2dd-a3cd-4f09-ac3f-8eebf84dd821 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:44.766089 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2393ddc4-502e-4ecb-8abb-1d694d0c7a24 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:44.804393 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:44.874250 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2393ddc4-502e-4ecb-8abb-1d694d0c7a24 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:44.907117 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f88e2dd-a3cd-4f09-ac3f-8eebf84dd821 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:44.912841 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86856d0b-a850-4389-bf81-a3cf8097272c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:44.913656 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86856d0b-a850-4389-bf81-a3cf8097272c None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:44.914030 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86856d0b-a850-4389-bf81-a3cf8097272c None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:44.914398 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86856d0b-a850-4389-bf81-a3cf8097272c None admin] PATH_INFO: `/v3/services/compute` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:44.915088 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86856d0b-a850-4389-bf81-a3cf8097272c None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:44.916911 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86856d0b-a850-4389-bf81-a3cf8097272c None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:44.922742 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:44.926883 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-86856d0b-a850-4389-bf81-a3cf8097272c None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Aug 28 00:42:44.926883 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:44.926883 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 28 00:42:44.926883 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 28 00:42:44.926883 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 28 00:42:44.926883 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 28 00:42:44.926883 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 28 00:42:44.926883 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 28 00:42:44.926883 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Aug 28 00:42:44.926883 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:42:44.926883 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 28 00:42:44.926883 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:42:44.926883 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:44.926883 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:44.926883 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:44.926883 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:44.926883 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:44.927990 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:44.927990 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:44.927990 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:44.927990 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:44.927990 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:44.927990 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:44.927990 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 28 00:42:44.927990 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 28 00:42:44.927990 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 28 00:42:44.927990 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 28 00:42:44.927990 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:44.927990 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:44.927990 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:44.927990 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:44.927990 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:44.927990 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:44.928828 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:44.928828 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:44.928828 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:44.928828 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:44.928828 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:44.928828 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:44.928828 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:44.928828 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:44.928828 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:44.928828 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:44.928828 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 28 00:42:44.928828 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 28 00:42:44.928828 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Aug 28 00:42:44.928828 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:42:44.933240 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 483/963] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Aug 28 00:42:44 2022] GET /identity/v3/services/compute => generated 88 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 28 00:42:44.934666 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86856d0b-a850-4389-bf81-a3cf8097272c None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:44.959360 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c166131-dce2-43d3-9111-6b9e877cbfaf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:44.960569 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c166131-dce2-43d3-9111-6b9e877cbfaf None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:44.961096 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c166131-dce2-43d3-9111-6b9e877cbfaf None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:44.961570 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c166131-dce2-43d3-9111-6b9e877cbfaf None admin] PATH_INFO: `/v3/domains` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:44.963087 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c166131-dce2-43d3-9111-6b9e877cbfaf None admin] RBAC: Adding query filter params (name=Default) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:44.963698 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c166131-dce2-43d3-9111-6b9e877cbfaf None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:44.965680 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c166131-dce2-43d3-9111-6b9e877cbfaf None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:44.984523 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 481/964] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Aug 28 00:42:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:44.989879 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c166131-dce2-43d3-9111-6b9e877cbfaf None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:45.060902 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:45.064564 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86856d0b-a850-4389-bf81-a3cf8097272c None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:45.111342 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b91558ca-f06d-465f-bd2d-c36c85f4101a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:45.113022 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b91558ca-f06d-465f-bd2d-c36c85f4101a None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:45.113531 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b91558ca-f06d-465f-bd2d-c36c85f4101a None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:45.114208 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b91558ca-f06d-465f-bd2d-c36c85f4101a None admin] PATH_INFO: `/v3/services` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:45.115044 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b91558ca-f06d-465f-bd2d-c36c85f4101a None admin] RBAC: Adding query filter params (name=compute) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:45.115709 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b91558ca-f06d-465f-bd2d-c36c85f4101a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:45.118543 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b91558ca-f06d-465f-bd2d-c36c85f4101a None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:45.119591 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c166131-dce2-43d3-9111-6b9e877cbfaf None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:45.126790 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 484/965] 10.222.0.42 () {56 vars in 1112 bytes} [Sun Aug 28 00:42:44 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:45.132753 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b91558ca-f06d-465f-bd2d-c36c85f4101a None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:45.163438 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a050345-5b27-44b5-88a0-951ed77679f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:45.164287 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a050345-5b27-44b5-88a0-951ed77679f1 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:45.164683 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a050345-5b27-44b5-88a0-951ed77679f1 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:45.165081 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a050345-5b27-44b5-88a0-951ed77679f1 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:45.173498 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a050345-5b27-44b5-88a0-951ed77679f1 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:45.175235 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a050345-5b27-44b5-88a0-951ed77679f1 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:45.188500 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-1a050345-5b27-44b5-88a0-951ed77679f1 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 28 00:42:45.188500 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:45.188500 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:45.188500 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:45.188500 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:45.188500 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:45.188500 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:45.188500 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:45.188500 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:45.188500 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:45.188500 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:45.188500 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:45.188500 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 28 00:42:45.188500 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 28 00:42:45.188500 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 28 00:42:45.188500 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 28 00:42:45.189809 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:45.189809 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:45.189809 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 28 00:42:45.189809 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 28 00:42:45.189809 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 28 00:42:45.189809 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 28 00:42:45.189809 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:45.189809 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:45.189809 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:45.189809 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:45.189809 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:45.189809 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:45.189809 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:45.189809 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:45.189809 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:45.189809 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:45.189809 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:45.189809 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:45.191037 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:45.191037 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:45.191037 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 28 00:42:45.191037 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 28 00:42:45.191037 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 28 00:42:45.191037 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 28 00:42:45.191037 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 28 00:42:45.191037 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 28 00:42:45.191037 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Aug 28 00:42:45.191037 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:45.193001 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 482/966] 10.222.0.42 () {56 vars in 1117 bytes} [Sun Aug 28 00:42:44 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:45.198403 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a050345-5b27-44b5-88a0-951ed77679f1 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:45.233914 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:45.275950 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b91558ca-f06d-465f-bd2d-c36c85f4101a None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:45.313150 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a050345-5b27-44b5-88a0-951ed77679f1 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:45.321748 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e5eeba1-4962-45e0-8319-dc6f2fd9861f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:45.322376 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e5eeba1-4962-45e0-8319-dc6f2fd9861f None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:45.322657 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e5eeba1-4962-45e0-8319-dc6f2fd9861f None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:45.322929 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e5eeba1-4962-45e0-8319-dc6f2fd9861f None admin] PATH_INFO: `/v3/services` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:45.323565 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e5eeba1-4962-45e0-8319-dc6f2fd9861f None admin] RBAC: Adding query filter params (type=compute) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:45.323866 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e5eeba1-4962-45e0-8319-dc6f2fd9861f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:45.325767 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e5eeba1-4962-45e0-8319-dc6f2fd9861f None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:45.333261 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 485/967] 10.222.0.42 () {56 vars in 1112 bytes} [Sun Aug 28 00:42:45 2022] GET /identity/v3/services?type=compute => generated 124 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 28 00:42:45.359582 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a98694cc-e2cb-4396-a9e5-9261cbd14e00 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:45.360769 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a98694cc-e2cb-4396-a9e5-9261cbd14e00 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:45.361300 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a98694cc-e2cb-4396-a9e5-9261cbd14e00 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:45.361778 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a98694cc-e2cb-4396-a9e5-9261cbd14e00 None admin] PATH_INFO: `/v3/users` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:45.362765 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a98694cc-e2cb-4396-a9e5-9261cbd14e00 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:45.363306 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a98694cc-e2cb-4396-a9e5-9261cbd14e00 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:45.365160 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a98694cc-e2cb-4396-a9e5-9261cbd14e00 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:45.379450 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c63991d0-faf1-47cb-9bad-d3f6870de956 None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:45.380162 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c63991d0-faf1-47cb-9bad-d3f6870de956 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:45.380672 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c63991d0-faf1-47cb-9bad-d3f6870de956 None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:45.382698 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 486/968] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:42:45 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 28 00:42:45.388025 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 483/969] 10.222.0.42 () {56 vars in 1149 bytes} [Sun Aug 28 00:42:45 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 28 00:42:45.389467 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-616e62fc-5188-422c-b888-185b2fc076d4 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:45.390007 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-616e62fc-5188-422c-b888-185b2fc076d4 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:45.390489 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-616e62fc-5188-422c-b888-185b2fc076d4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:45.393971 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a98694cc-e2cb-4396-a9e5-9261cbd14e00 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:45.411566 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-616e62fc-5188-422c-b888-185b2fc076d4 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:42:45.413208 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-616e62fc-5188-422c-b888-185b2fc076d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:42:45.441576 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:45.494637 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:45.497057 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 487/970] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:42:45 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 28 00:42:45.510040 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a98694cc-e2cb-4396-a9e5-9261cbd14e00 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:45.547580 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26be0849-a978-4f52-b0df-53b4151c70a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:45.548160 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26be0849-a978-4f52-b0df-53b4151c70a0 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:45.548431 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26be0849-a978-4f52-b0df-53b4151c70a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:45.548741 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26be0849-a978-4f52-b0df-53b4151c70a0 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:45.625294 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26be0849-a978-4f52-b0df-53b4151c70a0 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:45.626926 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26be0849-a978-4f52-b0df-53b4151c70a0 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:45.719990 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:45.739406 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 484/971] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Aug 28 00:42:45 2022] GET /identity/v3/auth/tokens => generated 1474 bytes in 346 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Aug 28 00:42:45.744952 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-616e62fc-5188-422c-b888-185b2fc076d4 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:45.771152 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26be0849-a978-4f52-b0df-53b4151c70a0 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:45.852183 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-616e62fc-5188-422c-b888-185b2fc076d4 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:45.882751 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26be0849-a978-4f52-b0df-53b4151c70a0 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:45.890236 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0b5f29a-1430-40cf-a755-fffd6bbd6223 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:45.890880 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0b5f29a-1430-40cf-a755-fffd6bbd6223 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:45.891155 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0b5f29a-1430-40cf-a755-fffd6bbd6223 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:45.891421 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0b5f29a-1430-40cf-a755-fffd6bbd6223 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:45.903071 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0b5f29a-1430-40cf-a755-fffd6bbd6223 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:45.905106 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0b5f29a-1430-40cf-a755-fffd6bbd6223 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:45.921562 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-b0b5f29a-1430-40cf-a755-fffd6bbd6223 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:42:45.921562 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:45.921562 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:45.921562 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:45.921562 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:45.921562 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:45.921562 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:45.921562 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:45.921562 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:45.921562 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:45.921562 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:45.921562 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:45.921562 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 28 00:42:45.921562 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 28 00:42:45.921562 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 28 00:42:45.921562 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 28 00:42:45.922833 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:45.922833 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:45.922833 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:45.922833 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:45.922833 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:45.922833 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:45.922833 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:45.922833 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:45.922833 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:45.922833 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:45.922833 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:45.922833 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:45.922833 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:45.922833 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:45.922833 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:45.922833 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:45.922833 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 28 00:42:45.923969 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 28 00:42:45.923969 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:42:45.923969 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:42:45.926721 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d561a098-fea8-4518-ba16-066749dc98b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:45.927531 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 488/972] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:42:45 2022] GET /identity/v3/domains/Default => generated 87 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:45.928390 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d561a098-fea8-4518-ba16-066749dc98b4 None admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:45.928520 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0b5f29a-1430-40cf-a755-fffd6bbd6223 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:45.929064 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d561a098-fea8-4518-ba16-066749dc98b4 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:45.929485 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d561a098-fea8-4518-ba16-066749dc98b4 None admin] PATH_INFO: `/v3/users` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:45.930356 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d561a098-fea8-4518-ba16-066749dc98b4 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:45.932111 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d561a098-fea8-4518-ba16-066749dc98b4 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:45.943054 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:45.950264 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.sql.core [None req-d561a098-fea8-4518-ba16-066749dc98b4 None admin] Conflict user: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Aug 28 00:42:45.950264 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Aug 28 00:42:45.950264 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [parameters: {'user_id': '59bc80aff6f442e6afc1fbcc407a55f1', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Aug 28 00:42:45.950264 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81507) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Aug 28 00:42:45.958466 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-d561a098-fea8-4518-ba16-066749dc98b4 None admin] Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default.: keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Aug 28 00:42:45.958466 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:45.958466 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Aug 28 00:42:45.958466 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application self.dialect.do_execute( Aug 28 00:42:45.958466 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Aug 28 00:42:45.958466 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Aug 28 00:42:45.958466 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Aug 28 00:42:45.958466 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application result = self._query(query) Aug 28 00:42:45.958466 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Aug 28 00:42:45.958466 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application conn.query(q) Aug 28 00:42:45.958466 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Aug 28 00:42:45.958466 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Aug 28 00:42:45.958466 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Aug 28 00:42:45.958466 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application result.read() Aug 28 00:42:45.958466 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Aug 28 00:42:45.959611 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Aug 28 00:42:45.959611 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Aug 28 00:42:45.959611 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application packet.raise_for_error() Aug 28 00:42:45.959611 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Aug 28 00:42:45.959611 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Aug 28 00:42:45.959611 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Aug 28 00:42:45.959611 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Aug 28 00:42:45.959611 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Aug 28 00:42:45.959611 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:45.959611 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Aug 28 00:42:45.959611 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:45.959611 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:45.959611 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Aug 28 00:42:45.959611 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return method(*args, **kwargs) Aug 28 00:42:45.959611 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 141, in create_user Aug 28 00:42:45.959611 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return base.filter_user(user_ref.to_dict()) Aug 28 00:42:45.959611 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Aug 28 00:42:45.959611 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application next(self.gen) Aug 28 00:42:45.960772 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Aug 28 00:42:45.960772 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application yield resource Aug 28 00:42:45.960772 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Aug 28 00:42:45.960772 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application next(self.gen) Aug 28 00:42:45.960772 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Aug 28 00:42:45.960772 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application self.session.rollback() Aug 28 00:42:45.960772 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Aug 28 00:42:45.960772 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application self.force_reraise() Aug 28 00:42:45.960772 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Aug 28 00:42:45.960772 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise self.value Aug 28 00:42:45.960772 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Aug 28 00:42:45.960772 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Aug 28 00:42:45.960772 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Aug 28 00:42:45.960772 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application session.commit() Aug 28 00:42:45.960772 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Aug 28 00:42:45.960772 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Aug 28 00:42:45.960772 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Aug 28 00:42:45.961741 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application self._prepare_impl() Aug 28 00:42:45.961741 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Aug 28 00:42:45.961741 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application self.session.flush() Aug 28 00:42:45.961741 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Aug 28 00:42:45.961741 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application self._flush(objects) Aug 28 00:42:45.961741 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Aug 28 00:42:45.961741 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Aug 28 00:42:45.961741 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Aug 28 00:42:45.961741 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application compat.raise_( Aug 28 00:42:45.961741 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Aug 28 00:42:45.961741 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception Aug 28 00:42:45.961741 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Aug 28 00:42:45.961741 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application flush_context.execute() Aug 28 00:42:45.961741 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Aug 28 00:42:45.961741 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rec.execute(self) Aug 28 00:42:45.961741 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Aug 28 00:42:45.961741 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Aug 28 00:42:45.962791 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Aug 28 00:42:45.962791 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application _emit_insert_statements( Aug 28 00:42:45.962791 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1238, in _emit_insert_statements Aug 28 00:42:45.962791 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application result = connection._execute_20( Aug 28 00:42:45.962791 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Aug 28 00:42:45.962791 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Aug 28 00:42:45.962791 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Aug 28 00:42:45.962791 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Aug 28 00:42:45.962791 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Aug 28 00:42:45.962791 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ret = self._execute_context( Aug 28 00:42:45.962791 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Aug 28 00:42:45.962791 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Aug 28 00:42:45.962791 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Aug 28 00:42:45.962791 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Aug 28 00:42:45.962791 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Aug 28 00:42:45.962791 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception Aug 28 00:42:45.963767 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Aug 28 00:42:45.963767 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application self.dialect.do_execute( Aug 28 00:42:45.963767 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Aug 28 00:42:45.963767 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Aug 28 00:42:45.963767 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Aug 28 00:42:45.963767 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application result = self._query(query) Aug 28 00:42:45.963767 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Aug 28 00:42:45.963767 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application conn.query(q) Aug 28 00:42:45.963767 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Aug 28 00:42:45.963767 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Aug 28 00:42:45.963767 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Aug 28 00:42:45.963767 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application result.read() Aug 28 00:42:45.963767 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Aug 28 00:42:45.963767 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Aug 28 00:42:45.963767 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Aug 28 00:42:45.963767 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application packet.raise_for_error() Aug 28 00:42:45.964609 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Aug 28 00:42:45.964609 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Aug 28 00:42:45.964609 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Aug 28 00:42:45.964609 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Aug 28 00:42:45.964609 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Aug 28 00:42:45.964609 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Aug 28 00:42:45.964609 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [parameters: {'user_id': '59bc80aff6f442e6afc1fbcc407a55f1', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Aug 28 00:42:45.964609 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Aug 28 00:42:45.964609 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:45.964609 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 28 00:42:45.964609 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:45.964609 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:45.964609 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:45.964609 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:45.964609 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:45.965617 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:45.965617 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:45.965617 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:45.965617 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:45.965617 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:45.965617 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:45.965617 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:45.965617 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 244, in post Aug 28 00:42:45.965617 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.create_user( Aug 28 00:42:45.965617 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:45.965617 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:45.965617 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 28 00:42:45.965617 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 28 00:42:45.965617 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 28 00:42:45.965617 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 28 00:42:45.965617 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1000, in create_user Aug 28 00:42:45.965617 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ref = self._create_user_with_federated_objects(user, driver) Aug 28 00:42:45.966648 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 966, in _create_user_with_federated_objects Aug 28 00:42:45.966648 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application user = driver.create_user(user['id'], user) Aug 28 00:42:45.966648 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Aug 28 00:42:45.966648 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Aug 28 00:42:45.966648 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Aug 28 00:42:45.966648 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:45.968876 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 485/973] 10.222.0.42 () {60 vars in 1131 bytes} [Sun Aug 28 00:42:45 2022] POST /identity/v3/users => generated 160 bytes in 198 msecs (HTTP/1.1 409) 5 headers in 183 bytes (2 switches on core 0) Aug 28 00:42:45.976990 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d561a098-fea8-4518-ba16-066749dc98b4 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:46.046173 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0b5f29a-1430-40cf-a755-fffd6bbd6223 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:46.051641 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:46.092047 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-050bdf97-42b7-4fa4-a73c-4b41b7957be4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:46.092986 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-050bdf97-42b7-4fa4-a73c-4b41b7957be4 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:46.093322 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-050bdf97-42b7-4fa4-a73c-4b41b7957be4 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:46.093610 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-050bdf97-42b7-4fa4-a73c-4b41b7957be4 None admin] PATH_INFO: `/v3/domains` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:46.094352 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-050bdf97-42b7-4fa4-a73c-4b41b7957be4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:46.094717 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-050bdf97-42b7-4fa4-a73c-4b41b7957be4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:46.096188 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-050bdf97-42b7-4fa4-a73c-4b41b7957be4 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:46.117672 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 489/974] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Aug 28 00:42:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:46.124222 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-050bdf97-42b7-4fa4-a73c-4b41b7957be4 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:46.126040 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d561a098-fea8-4518-ba16-066749dc98b4 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:46.165181 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5c4f115-e307-4107-9849-8ed086a54359 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:46.165822 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5c4f115-e307-4107-9849-8ed086a54359 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:46.166139 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5c4f115-e307-4107-9849-8ed086a54359 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:46.166427 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5c4f115-e307-4107-9849-8ed086a54359 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:46.175061 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5c4f115-e307-4107-9849-8ed086a54359 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:46.176826 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5c4f115-e307-4107-9849-8ed086a54359 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:46.191622 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-e5c4f115-e307-4107-9849-8ed086a54359 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Aug 28 00:42:46.191622 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:46.191622 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:46.191622 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:46.191622 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:46.191622 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:46.191622 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:46.191622 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:46.191622 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:46.191622 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:46.191622 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:46.191622 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:46.191622 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 28 00:42:46.191622 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 28 00:42:46.191622 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 28 00:42:46.191622 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 28 00:42:46.192671 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:46.192671 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:46.192671 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 28 00:42:46.192671 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 28 00:42:46.192671 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 28 00:42:46.192671 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 28 00:42:46.192671 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:46.192671 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:46.192671 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:46.192671 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:46.192671 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:46.192671 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:46.192671 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:46.192671 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:46.192671 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:46.192671 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:46.192671 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:46.192671 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:46.193649 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:46.193649 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:46.193649 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 28 00:42:46.193649 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 28 00:42:46.193649 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 28 00:42:46.193649 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 28 00:42:46.193649 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 28 00:42:46.193649 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 28 00:42:46.193649 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Aug 28 00:42:46.193649 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:46.197453 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 486/975] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Aug 28 00:42:45 2022] GET /identity/v3/users/cinder => generated 84 bytes in 222 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:46.199088 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5c4f115-e307-4107-9849-8ed086a54359 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:46.225939 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:46.273653 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-050bdf97-42b7-4fa4-a73c-4b41b7957be4 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:46.319824 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9f5fe49-2126-4d09-ad92-8874a2b274ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:46.320725 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f5fe49-2126-4d09-ad92-8874a2b274ef None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:46.321043 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f5fe49-2126-4d09-ad92-8874a2b274ef None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:46.321320 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f5fe49-2126-4d09-ad92-8874a2b274ef None admin] PATH_INFO: `/v3/projects/service` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:46.326185 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5c4f115-e307-4107-9849-8ed086a54359 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:46.331833 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9f5fe49-2126-4d09-ad92-8874a2b274ef None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:46.334395 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9f5fe49-2126-4d09-ad92-8874a2b274ef None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:46.340264 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:46.352355 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-c9f5fe49-2126-4d09-ad92-8874a2b274ef None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 28 00:42:46.352355 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:46.352355 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:46.352355 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:46.352355 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:46.352355 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:46.352355 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:46.352355 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:46.352355 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:46.352355 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:46.352355 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:46.352355 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:46.352355 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 28 00:42:46.352355 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 28 00:42:46.352355 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 28 00:42:46.353344 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 28 00:42:46.353344 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:46.353344 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:46.353344 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:46.353344 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:46.353344 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:46.353344 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:46.353344 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:46.353344 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:46.353344 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:46.353344 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:46.353344 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:46.353344 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:46.353344 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:46.353344 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:46.353344 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:46.353344 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:46.354270 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 28 00:42:46.354270 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 28 00:42:46.354270 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 28 00:42:46.354270 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 28 00:42:46.354270 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 28 00:42:46.354270 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 28 00:42:46.354270 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 28 00:42:46.354270 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:42:46.357394 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 490/976] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Aug 28 00:42:46 2022] GET /identity/v3/projects/service => generated 88 bytes in 234 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:46.360360 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9f5fe49-2126-4d09-ad92-8874a2b274ef None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:46.378966 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5db78240-1a8a-4ee4-91cc-3766f78d3af7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:46.379663 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5db78240-1a8a-4ee4-91cc-3766f78d3af7 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:46.379991 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5db78240-1a8a-4ee4-91cc-3766f78d3af7 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:46.380267 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5db78240-1a8a-4ee4-91cc-3766f78d3af7 None admin] PATH_INFO: `/v3/users` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:46.381672 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5db78240-1a8a-4ee4-91cc-3766f78d3af7 None admin] RBAC: Adding query filter params (name=cinder) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:46.381982 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5db78240-1a8a-4ee4-91cc-3766f78d3af7 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:46.383788 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5db78240-1a8a-4ee4-91cc-3766f78d3af7 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:46.436290 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 487/977] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Aug 28 00:42:46 2022] GET /identity/v3/users?name=cinder => generated 354 bytes in 238 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 28 00:42:46.466422 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12673070-eb36-4d4b-90e3-31e0bbc3f51f None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:46.467118 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12673070-eb36-4d4b-90e3-31e0bbc3f51f None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:46.467603 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12673070-eb36-4d4b-90e3-31e0bbc3f51f None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:46.469732 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 488/978] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:42:46 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 28 00:42:46.479455 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89c2f0e1-d2b2-478f-9722-360b461eef5c None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:46.480187 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89c2f0e1-d2b2-478f-9722-360b461eef5c None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:46.480741 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89c2f0e1-d2b2-478f-9722-360b461eef5c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:46.504707 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-89c2f0e1-d2b2-478f-9722-360b461eef5c None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:42:46.507116 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-89c2f0e1-d2b2-478f-9722-360b461eef5c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:42:46.510320 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9f5fe49-2126-4d09-ad92-8874a2b274ef None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:46.543751 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:46.551859 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fd912db-aeae-478a-9b4a-fa95a2a33300 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:46.552761 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fd912db-aeae-478a-9b4a-fa95a2a33300 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:46.553229 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fd912db-aeae-478a-9b4a-fa95a2a33300 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:46.553633 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fd912db-aeae-478a-9b4a-fa95a2a33300 None admin] PATH_INFO: `/v3/projects` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:46.554744 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fd912db-aeae-478a-9b4a-fa95a2a33300 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:46.555356 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fd912db-aeae-478a-9b4a-fa95a2a33300 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:46.557651 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fd912db-aeae-478a-9b4a-fa95a2a33300 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:46.579512 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 491/979] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Aug 28 00:42:46 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:46.587682 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fd912db-aeae-478a-9b4a-fa95a2a33300 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:46.603977 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 489/980] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:42:46 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 28 00:42:46.700112 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fd912db-aeae-478a-9b4a-fa95a2a33300 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:46.736762 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb323222-3f7d-4343-b4f2-a3a66eb39846 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:46.737407 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb323222-3f7d-4343-b4f2-a3a66eb39846 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:46.737682 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb323222-3f7d-4343-b4f2-a3a66eb39846 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:46.737946 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb323222-3f7d-4343-b4f2-a3a66eb39846 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:46.738680 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb323222-3f7d-4343-b4f2-a3a66eb39846 None admin] RBAC: Adding query filter params (role.id=2c6107d249594bd3abf791034b71e98f, scope.project.id=f53e6fdb369b480fbebb4086a505df67, user.id=eede08f020e04282bb994b855e371433) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:46.738978 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb323222-3f7d-4343-b4f2-a3a66eb39846 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:46.740530 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb323222-3f7d-4343-b4f2-a3a66eb39846 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:46.747467 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 492/981] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Aug 28 00:42:46 2022] GET /identity/v3/role_assignments?user.id=eede08f020e04282bb994b855e371433&scope.project.id=f53e6fdb369b480fbebb4086a505df67&role.id=2c6107d249594bd3abf791034b71e98f => generated 608 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:46.876836 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89c2f0e1-d2b2-478f-9722-360b461eef5c None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:46.980064 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:46.984725 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89c2f0e1-d2b2-478f-9722-360b461eef5c None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:47.027303 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b6c8498-67e2-4937-b1d9-38711b40edd8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:47.028253 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b6c8498-67e2-4937-b1d9-38711b40edd8 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:47.028694 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b6c8498-67e2-4937-b1d9-38711b40edd8 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:47.029219 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b6c8498-67e2-4937-b1d9-38711b40edd8 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:47.029945 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b6c8498-67e2-4937-b1d9-38711b40edd8 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:47.031445 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b6c8498-67e2-4937-b1d9-38711b40edd8 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:47.041102 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-3b6c8498-67e2-4937-b1d9-38711b40edd8 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 28 00:42:47.041102 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:47.041102 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 28 00:42:47.041102 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 28 00:42:47.041102 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 28 00:42:47.041102 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 28 00:42:47.041102 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 28 00:42:47.041102 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 28 00:42:47.041102 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 28 00:42:47.041102 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:47.041102 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 28 00:42:47.041102 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:47.041102 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:47.041102 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:47.041102 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:47.041102 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:47.041102 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:47.042285 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:47.042285 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:47.042285 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:47.042285 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:47.042285 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:47.042285 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:47.042285 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 28 00:42:47.042285 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 28 00:42:47.042285 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 28 00:42:47.042285 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 28 00:42:47.042285 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:47.042285 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:47.042285 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:47.042285 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:47.042285 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:47.042285 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:47.043132 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:47.043132 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:47.043132 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:47.043132 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:47.043132 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:47.043132 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:47.043132 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:47.043132 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:47.043132 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:47.043132 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:47.043132 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 28 00:42:47.043132 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 28 00:42:47.043132 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 28 00:42:47.043132 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:47.043867 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 490/982] 10.222.0.42 () {56 vars in 1126 bytes} [Sun Aug 28 00:42:46 2022] GET /identity/v3/services/object-store => generated 93 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:47.050145 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb323222-3f7d-4343-b4f2-a3a66eb39846 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:47.128692 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:47.184712 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb323222-3f7d-4343-b4f2-a3a66eb39846 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:47.227000 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df575588-4415-42e0-9ba1-9c1c04a68486 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:47.227538 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df575588-4415-42e0-9ba1-9c1c04a68486 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:47.227819 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df575588-4415-42e0-9ba1-9c1c04a68486 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:47.228089 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df575588-4415-42e0-9ba1-9c1c04a68486 None admin] PATH_INFO: `/v3/services` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:47.228737 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df575588-4415-42e0-9ba1-9c1c04a68486 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:47.229074 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df575588-4415-42e0-9ba1-9c1c04a68486 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:47.231864 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df575588-4415-42e0-9ba1-9c1c04a68486 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:47.239540 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 493/983] 10.222.0.42 () {56 vars in 1122 bytes} [Sun Aug 28 00:42:47 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:47.245895 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b6c8498-67e2-4937-b1d9-38711b40edd8 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:47.335177 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6183a01-2906-437f-a30e-6d89a6b38e8d None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:47.335529 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6183a01-2906-437f-a30e-6d89a6b38e8d None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:47.335727 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6183a01-2906-437f-a30e-6d89a6b38e8d None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:47.337242 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 494/984] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:42:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 28 00:42:47.342696 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-073b7863-f84a-4ca3-8910-0758478012c7 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:47.342974 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-073b7863-f84a-4ca3-8910-0758478012c7 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:47.343251 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-073b7863-f84a-4ca3-8910-0758478012c7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:47.361742 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-073b7863-f84a-4ca3-8910-0758478012c7 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:42:47.362965 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-073b7863-f84a-4ca3-8910-0758478012c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:42:47.371363 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b6c8498-67e2-4937-b1d9-38711b40edd8 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:47.414885 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a8ade09-53a4-4c9f-bc00-0c8dcc352287 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:47.415501 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a8ade09-53a4-4c9f-bc00-0c8dcc352287 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:47.415778 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a8ade09-53a4-4c9f-bc00-0c8dcc352287 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:47.416044 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a8ade09-53a4-4c9f-bc00-0c8dcc352287 None admin] PATH_INFO: `/v3/services` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:47.416678 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a8ade09-53a4-4c9f-bc00-0c8dcc352287 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:47.417002 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a8ade09-53a4-4c9f-bc00-0c8dcc352287 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:47.418396 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a8ade09-53a4-4c9f-bc00-0c8dcc352287 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:47.424216 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 491/985] 10.222.0.42 () {56 vars in 1122 bytes} [Sun Aug 28 00:42:47 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 28 00:42:47.430603 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a8ade09-53a4-4c9f-bc00-0c8dcc352287 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:47.450381 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 495/986] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:42:47 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 28 00:42:47.467715 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:47.550305 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a8ade09-53a4-4c9f-bc00-0c8dcc352287 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:47.599621 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3a1474b-2b69-4d9c-881a-dd1f0d062ce7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:47.600643 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3a1474b-2b69-4d9c-881a-dd1f0d062ce7 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:47.601168 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3a1474b-2b69-4d9c-881a-dd1f0d062ce7 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:47.601641 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3a1474b-2b69-4d9c-881a-dd1f0d062ce7 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:47.602893 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3a1474b-2b69-4d9c-881a-dd1f0d062ce7 None admin] RBAC: Adding query filter params (interface=admin, service_id=2ba2936a2af44b56a45e060abc738be4, region_id=RegionOne) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:47.603409 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3a1474b-2b69-4d9c-881a-dd1f0d062ce7 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:47.605992 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3a1474b-2b69-4d9c-881a-dd1f0d062ce7 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:47.623852 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 492/987] 10.222.0.42 () {56 vars in 1249 bytes} [Sun Aug 28 00:42:47 2022] GET /identity/v3/endpoints?service_id=2ba2936a2af44b56a45e060abc738be4&interface=admin®ion_id=RegionOne => generated 193 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:47.625630 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-073b7863-f84a-4ca3-8910-0758478012c7 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:47.716916 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:47.738507 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3a1474b-2b69-4d9c-881a-dd1f0d062ce7 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:47.806246 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-073b7863-f84a-4ca3-8910-0758478012c7 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:47.864408 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c84a00c8-3a15-40cd-a67e-9196c850be7e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:47.865482 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c84a00c8-3a15-40cd-a67e-9196c850be7e None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:47.865936 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c84a00c8-3a15-40cd-a67e-9196c850be7e None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:47.866379 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c84a00c8-3a15-40cd-a67e-9196c850be7e None admin] PATH_INFO: `/v3/services` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:47.867405 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c84a00c8-3a15-40cd-a67e-9196c850be7e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:47.870724 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c84a00c8-3a15-40cd-a67e-9196c850be7e None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:47.883493 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 496/988] 10.222.0.42 () {56 vars in 1087 bytes} [Sun Aug 28 00:42:47 2022] GET /identity/v3/services => generated 1497 bytes in 259 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 00:42:47.888857 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:47.902421 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3a1474b-2b69-4d9c-881a-dd1f0d062ce7 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:47.945573 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-700de5eb-7528-4095-ae37-02e2e6c794f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:47.946589 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-700de5eb-7528-4095-ae37-02e2e6c794f6 None admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:47.947089 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-700de5eb-7528-4095-ae37-02e2e6c794f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:47.947569 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-700de5eb-7528-4095-ae37-02e2e6c794f6 None admin] PATH_INFO: `/v3/services` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:47.948445 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-700de5eb-7528-4095-ae37-02e2e6c794f6 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:47.950103 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-700de5eb-7528-4095-ae37-02e2e6c794f6 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:47.969516 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 493/989] 10.222.0.42 () {60 vars in 1141 bytes} [Sun Aug 28 00:42:47 2022] POST /identity/v3/services => generated 242 bytes in 232 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:42:48.709076 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d2159f-8ee2-4282-9624-9e722ffeec6e None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:48.709394 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d2159f-8ee2-4282-9624-9e722ffeec6e None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:48.709531 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d2159f-8ee2-4282-9624-9e722ffeec6e None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:48.711185 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 497/990] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:42:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 28 00:42:48.718106 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ad34d6e-4c7f-4df7-9ac4-aea3aeca51a8 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:48.718644 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ad34d6e-4c7f-4df7-9ac4-aea3aeca51a8 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:48.719120 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ad34d6e-4c7f-4df7-9ac4-aea3aeca51a8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:48.749028 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-7ad34d6e-4c7f-4df7-9ac4-aea3aeca51a8 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:42:48.750995 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-7ad34d6e-4c7f-4df7-9ac4-aea3aeca51a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:42:48.881914 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 494/991] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:42:48 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 28 00:42:49.244872 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28d2159f-8ee2-4282-9624-9e722ffeec6e None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:49.368218 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28d2159f-8ee2-4282-9624-9e722ffeec6e None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:49.385801 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:49.413685 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d15b02b-a1b3-4549-837b-9811ada38367 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:49.414513 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d15b02b-a1b3-4549-837b-9811ada38367 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:49.414924 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d15b02b-a1b3-4549-837b-9811ada38367 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:49.415337 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d15b02b-a1b3-4549-837b-9811ada38367 None admin] PATH_INFO: `/v3/services/image` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:49.416116 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d15b02b-a1b3-4549-837b-9811ada38367 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:49.417783 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d15b02b-a1b3-4549-837b-9811ada38367 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:49.428467 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-0d15b02b-a1b3-4549-837b-9811ada38367 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Aug 28 00:42:49.428467 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:49.428467 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 28 00:42:49.428467 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 28 00:42:49.428467 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 28 00:42:49.428467 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 28 00:42:49.428467 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 28 00:42:49.428467 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 28 00:42:49.428467 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Aug 28 00:42:49.428467 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:42:49.428467 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 28 00:42:49.428467 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:42:49.428467 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:49.428467 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:49.428467 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:49.428467 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:49.428467 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:49.430480 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:49.430480 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:49.430480 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:49.430480 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:49.430480 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:49.430480 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:49.430480 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 28 00:42:49.430480 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 28 00:42:49.430480 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 28 00:42:49.430480 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 28 00:42:49.430480 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:49.430480 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:49.430480 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:49.430480 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:49.430480 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:49.430480 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:49.432101 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:49.432101 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:49.432101 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:49.432101 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:49.432101 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:49.432101 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:49.432101 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:49.432101 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:49.432101 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:49.432101 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:49.432101 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 28 00:42:49.432101 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 28 00:42:49.432101 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Aug 28 00:42:49.432101 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:42:49.437365 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 498/992] 10.222.0.42 () {56 vars in 1105 bytes} [Sun Aug 28 00:42:49 2022] GET /identity/v3/services/image => generated 86 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:49.438983 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ad34d6e-4c7f-4df7-9ac4-aea3aeca51a8 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:49.579841 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:49.628228 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ad34d6e-4c7f-4df7-9ac4-aea3aeca51a8 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:49.691911 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8ba1d9e-bd15-4d46-a3db-e872d72ceb68 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:49.692816 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8ba1d9e-bd15-4d46-a3db-e872d72ceb68 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:49.693401 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8ba1d9e-bd15-4d46-a3db-e872d72ceb68 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:49.693884 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8ba1d9e-bd15-4d46-a3db-e872d72ceb68 None admin] PATH_INFO: `/v3/services` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:49.695029 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8ba1d9e-bd15-4d46-a3db-e872d72ceb68 None admin] RBAC: Adding query filter params (name=image) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:49.695564 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8ba1d9e-bd15-4d46-a3db-e872d72ceb68 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:49.698237 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8ba1d9e-bd15-4d46-a3db-e872d72ceb68 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:49.708652 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 495/993] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:42:49 2022] GET /identity/v3/services?name=image => generated 122 bytes in 270 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 28 00:42:49.717783 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d15b02b-a1b3-4549-837b-9811ada38367 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:49.906056 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d15b02b-a1b3-4549-837b-9811ada38367 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:49.968481 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c40e4689-915f-4aa2-8ac6-ecc9efe8b1f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:49.969462 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c40e4689-915f-4aa2-8ac6-ecc9efe8b1f3 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:49.969940 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c40e4689-915f-4aa2-8ac6-ecc9efe8b1f3 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:49.970414 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c40e4689-915f-4aa2-8ac6-ecc9efe8b1f3 None admin] PATH_INFO: `/v3/services` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:49.971551 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c40e4689-915f-4aa2-8ac6-ecc9efe8b1f3 None admin] RBAC: Adding query filter params (type=image) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:49.972070 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c40e4689-915f-4aa2-8ac6-ecc9efe8b1f3 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:49.974528 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c40e4689-915f-4aa2-8ac6-ecc9efe8b1f3 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:49.982717 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 499/994] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:42:49 2022] GET /identity/v3/services?type=image => generated 122 bytes in 266 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:50.381640 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a39c229-7099-4435-856f-d9ad18c1da31 None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:50.382207 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a39c229-7099-4435-856f-d9ad18c1da31 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:50.382630 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a39c229-7099-4435-856f-d9ad18c1da31 None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:50.385403 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 496/995] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:42:50 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:42:50.391256 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b60231d4-e15a-4a28-8a49-1707d1f5c887 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:50.392002 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b60231d4-e15a-4a28-8a49-1707d1f5c887 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:50.392002 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b60231d4-e15a-4a28-8a49-1707d1f5c887 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:50.411628 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-b60231d4-e15a-4a28-8a49-1707d1f5c887 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:42:50.412837 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-b60231d4-e15a-4a28-8a49-1707d1f5c887 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:42:50.468081 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:50.532166 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 500/996] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:42:50 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 142 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 28 00:42:50.564992 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea14c25-57b2-40bc-a0f7-99d3f139736c None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:50.565285 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea14c25-57b2-40bc-a0f7-99d3f139736c None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:50.565585 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea14c25-57b2-40bc-a0f7-99d3f139736c None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:50.567079 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 497/997] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:42:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 28 00:42:50.572501 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d179e6ce-e73a-4d1f-b085-81ed086a72c6 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:50.572803 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d179e6ce-e73a-4d1f-b085-81ed086a72c6 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:50.573101 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d179e6ce-e73a-4d1f-b085-81ed086a72c6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:50.592872 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-d179e6ce-e73a-4d1f-b085-81ed086a72c6 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:42:50.594131 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-d179e6ce-e73a-4d1f-b085-81ed086a72c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:42:50.673492 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 501/998] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:42:50 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 28 00:42:50.794839 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bea14c25-57b2-40bc-a0f7-99d3f139736c None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:50.911792 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bea14c25-57b2-40bc-a0f7-99d3f139736c None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:50.937585 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d179e6ce-e73a-4d1f-b085-81ed086a72c6 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:50.954869 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2542463-241b-41a4-91ee-b4415560a8ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:50.955923 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2542463-241b-41a4-91ee-b4415560a8ad None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:50.956406 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2542463-241b-41a4-91ee-b4415560a8ad None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:50.956890 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2542463-241b-41a4-91ee-b4415560a8ad None admin] PATH_INFO: `/v3/services/compute` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:50.957952 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2542463-241b-41a4-91ee-b4415560a8ad None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:50.960404 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2542463-241b-41a4-91ee-b4415560a8ad None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:50.977258 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-a2542463-241b-41a4-91ee-b4415560a8ad None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Aug 28 00:42:50.977258 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:50.977258 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 28 00:42:50.977258 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 28 00:42:50.977258 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 28 00:42:50.977258 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 28 00:42:50.977258 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 28 00:42:50.977258 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 28 00:42:50.977258 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Aug 28 00:42:50.977258 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:50.977258 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 28 00:42:50.977258 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:50.977258 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:50.977258 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:50.977258 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:50.977258 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:50.977258 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:50.978373 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:50.978373 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:50.978373 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:50.978373 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:50.978373 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:50.978373 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:50.978373 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 28 00:42:50.978373 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 28 00:42:50.978373 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 28 00:42:50.978373 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 28 00:42:50.978373 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:50.978373 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:50.978373 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:50.978373 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:50.978373 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:50.978373 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:50.979225 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:50.979225 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:50.979225 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:50.979225 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:50.979225 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:50.979225 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:50.979225 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:50.979225 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:50.979225 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:50.979225 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:50.979225 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 28 00:42:50.979225 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 28 00:42:50.979225 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Aug 28 00:42:50.979225 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:50.979977 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 498/999] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Aug 28 00:42:50 2022] GET /identity/v3/services/compute => generated 88 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:50.986588 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2542463-241b-41a4-91ee-b4415560a8ad None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:50.992994 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:51.046686 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:51.050078 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d179e6ce-e73a-4d1f-b085-81ed086a72c6 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:51.093928 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f47617a-fa5d-422c-aa61-ea78dd200522 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:51.094663 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f47617a-fa5d-422c-aa61-ea78dd200522 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:51.095008 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f47617a-fa5d-422c-aa61-ea78dd200522 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:51.095278 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f47617a-fa5d-422c-aa61-ea78dd200522 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:51.095864 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f47617a-fa5d-422c-aa61-ea78dd200522 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:51.097263 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f47617a-fa5d-422c-aa61-ea78dd200522 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:51.106364 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-2f47617a-fa5d-422c-aa61-ea78dd200522 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 28 00:42:51.106364 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:51.106364 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 28 00:42:51.106364 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 28 00:42:51.106364 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 28 00:42:51.106364 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 28 00:42:51.106364 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 28 00:42:51.106364 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 28 00:42:51.106364 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 28 00:42:51.106364 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:42:51.106364 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 28 00:42:51.106364 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:42:51.106364 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:51.106364 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:51.106364 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:51.106364 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:51.106364 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:51.109205 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:51.109205 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:51.109205 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:51.109205 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:51.109205 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:51.109205 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:51.109205 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 28 00:42:51.109205 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 28 00:42:51.109205 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 28 00:42:51.109205 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 28 00:42:51.109205 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:51.109205 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:51.109205 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:51.109205 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:51.109205 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:51.109205 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:51.110123 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:51.110123 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:51.110123 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:51.110123 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:51.110123 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:51.110123 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:51.110123 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:51.110123 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:51.110123 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:51.110123 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:51.110123 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 28 00:42:51.110123 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 28 00:42:51.110123 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Aug 28 00:42:51.110123 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:42:51.110123 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 502/1000] 10.222.0.42 () {56 vars in 1126 bytes} [Sun Aug 28 00:42:50 2022] GET /identity/v3/services/object-store => generated 93 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 28 00:42:51.113584 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f47617a-fa5d-422c-aa61-ea78dd200522 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:51.158762 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2542463-241b-41a4-91ee-b4415560a8ad None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:51.205637 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0742b05d-ac3d-46dc-ab7c-7264d68e5bfe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:51.206370 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0742b05d-ac3d-46dc-ab7c-7264d68e5bfe None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:51.206710 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0742b05d-ac3d-46dc-ab7c-7264d68e5bfe None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:51.207045 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0742b05d-ac3d-46dc-ab7c-7264d68e5bfe None admin] PATH_INFO: `/v3/services` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:51.207998 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0742b05d-ac3d-46dc-ab7c-7264d68e5bfe None admin] RBAC: Adding query filter params (name=compute) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:51.208381 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0742b05d-ac3d-46dc-ab7c-7264d68e5bfe None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:51.210023 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0742b05d-ac3d-46dc-ab7c-7264d68e5bfe None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:51.216173 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 499/1001] 10.222.0.42 () {56 vars in 1112 bytes} [Sun Aug 28 00:42:50 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 231 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:51.223189 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0742b05d-ac3d-46dc-ab7c-7264d68e5bfe None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:51.228717 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f47617a-fa5d-422c-aa61-ea78dd200522 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:51.270588 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cc26a03-f319-4af4-a858-690ceb7eb209 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:51.271980 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc26a03-f319-4af4-a858-690ceb7eb209 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:51.271980 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc26a03-f319-4af4-a858-690ceb7eb209 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:51.271980 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc26a03-f319-4af4-a858-690ceb7eb209 None admin] PATH_INFO: `/v3/services` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:51.272370 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cc26a03-f319-4af4-a858-690ceb7eb209 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:51.272669 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cc26a03-f319-4af4-a858-690ceb7eb209 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:51.274084 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cc26a03-f319-4af4-a858-690ceb7eb209 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:51.284217 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 503/1002] 10.222.0.42 () {56 vars in 1122 bytes} [Sun Aug 28 00:42:51 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:51.285157 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cc26a03-f319-4af4-a858-690ceb7eb209 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:51.315755 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:51.318902 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:51.371543 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0742b05d-ac3d-46dc-ab7c-7264d68e5bfe None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:51.399479 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cc26a03-f319-4af4-a858-690ceb7eb209 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:51.430455 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eb678ac-094b-456c-85a0-ea06d101ee9c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:51.431114 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb678ac-094b-456c-85a0-ea06d101ee9c None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:51.431346 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb678ac-094b-456c-85a0-ea06d101ee9c None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:51.431610 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb678ac-094b-456c-85a0-ea06d101ee9c None admin] PATH_INFO: `/v3/services` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:51.432253 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eb678ac-094b-456c-85a0-ea06d101ee9c None admin] RBAC: Adding query filter params (type=compute) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:51.432541 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eb678ac-094b-456c-85a0-ea06d101ee9c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:51.433978 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eb678ac-094b-456c-85a0-ea06d101ee9c None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:51.446249 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 500/1003] 10.222.0.42 () {56 vars in 1112 bytes} [Sun Aug 28 00:42:51 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 28 00:42:51.448269 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eb678ac-094b-456c-85a0-ea06d101ee9c None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:51.449791 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b385d59-f9e0-4392-b109-8c16ba18d4d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:51.450744 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b385d59-f9e0-4392-b109-8c16ba18d4d2 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:51.451204 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b385d59-f9e0-4392-b109-8c16ba18d4d2 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:51.451603 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b385d59-f9e0-4392-b109-8c16ba18d4d2 None admin] PATH_INFO: `/v3/services` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:51.452577 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b385d59-f9e0-4392-b109-8c16ba18d4d2 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:51.453081 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b385d59-f9e0-4392-b109-8c16ba18d4d2 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:51.455367 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b385d59-f9e0-4392-b109-8c16ba18d4d2 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:51.468739 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 504/1004] 10.222.0.42 () {56 vars in 1122 bytes} [Sun Aug 28 00:42:51 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:51.470183 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b385d59-f9e0-4392-b109-8c16ba18d4d2 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:51.609518 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eb678ac-094b-456c-85a0-ea06d101ee9c None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:51.622453 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:51.637389 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b385d59-f9e0-4392-b109-8c16ba18d4d2 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:51.641652 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:51.652066 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f7a491c-299a-477b-aef4-374b0c2b1eb9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:51.652814 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f7a491c-299a-477b-aef4-374b0c2b1eb9 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:51.653258 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f7a491c-299a-477b-aef4-374b0c2b1eb9 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:51.653616 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f7a491c-299a-477b-aef4-374b0c2b1eb9 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:51.654618 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f7a491c-299a-477b-aef4-374b0c2b1eb9 None admin] RBAC: Adding query filter params (interface=public, service_id=93def5b7622a40a19550f3efea7a7b45, region_id=RegionOne) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:51.655030 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f7a491c-299a-477b-aef4-374b0c2b1eb9 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:51.657062 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f7a491c-299a-477b-aef4-374b0c2b1eb9 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:51.668086 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 501/1005] 10.222.0.42 () {56 vars in 1251 bytes} [Sun Aug 28 00:42:51 2022] GET /identity/v3/endpoints?service_id=93def5b7622a40a19550f3efea7a7b45&interface=public®ion_id=RegionOne => generated 194 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 28 00:42:51.669709 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f7a491c-299a-477b-aef4-374b0c2b1eb9 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:51.674965 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58f0e2e1-cfea-418a-b178-b5081c6140f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:51.675604 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58f0e2e1-cfea-418a-b178-b5081c6140f3 None admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:51.675883 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58f0e2e1-cfea-418a-b178-b5081c6140f3 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:51.676160 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58f0e2e1-cfea-418a-b178-b5081c6140f3 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:51.676859 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58f0e2e1-cfea-418a-b178-b5081c6140f3 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:51.678376 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58f0e2e1-cfea-418a-b178-b5081c6140f3 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:51.698779 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 505/1006] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 00:42:51 2022] POST /identity/v3/endpoints => generated 325 bytes in 229 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:42:51.879780 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f7a491c-299a-477b-aef4-374b0c2b1eb9 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:51.935490 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-158c0c4d-4c09-4911-b1b8-5632d61b0417 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:51.936103 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-158c0c4d-4c09-4911-b1b8-5632d61b0417 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:51.936403 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-158c0c4d-4c09-4911-b1b8-5632d61b0417 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:51.936673 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-158c0c4d-4c09-4911-b1b8-5632d61b0417 None admin] PATH_INFO: `/v3/services` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:51.937312 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-158c0c4d-4c09-4911-b1b8-5632d61b0417 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:51.938678 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-158c0c4d-4c09-4911-b1b8-5632d61b0417 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:51.946252 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 502/1007] 10.222.0.42 () {56 vars in 1087 bytes} [Sun Aug 28 00:42:51 2022] GET /identity/v3/services => generated 1727 bytes in 277 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 28 00:42:52.521174 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d06c8224-fe2d-40ce-866d-c8ec87971fe5 None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:52.521715 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d06c8224-fe2d-40ce-866d-c8ec87971fe5 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:52.522157 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d06c8224-fe2d-40ce-866d-c8ec87971fe5 None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:52.524622 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 506/1008] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:42:52 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:42:52.530580 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-967c4b85-fc5b-4fb5-a5c3-748bfb02eb1a None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:52.531339 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-967c4b85-fc5b-4fb5-a5c3-748bfb02eb1a None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:52.531339 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-967c4b85-fc5b-4fb5-a5c3-748bfb02eb1a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:52.550010 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-967c4b85-fc5b-4fb5-a5c3-748bfb02eb1a None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:42:52.551280 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-967c4b85-fc5b-4fb5-a5c3-748bfb02eb1a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:42:52.617390 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:52.639948 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 503/1009] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:42:52 2022] POST /identity/v3/auth/tokens => generated 1717 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 28 00:42:52.912027 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d06c8224-fe2d-40ce-866d-c8ec87971fe5 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:53.029208 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d06c8224-fe2d-40ce-866d-c8ec87971fe5 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:53.086238 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a16736b-247b-4230-a328-32f50e214d13 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:53.087287 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a16736b-247b-4230-a328-32f50e214d13 None admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:53.087775 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a16736b-247b-4230-a328-32f50e214d13 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:53.088271 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a16736b-247b-4230-a328-32f50e214d13 None admin] PATH_INFO: `/v3/services` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:53.089528 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a16736b-247b-4230-a328-32f50e214d13 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:53.092202 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a16736b-247b-4230-a328-32f50e214d13 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:53.129054 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 507/1010] 10.222.0.42 () {60 vars in 1141 bytes} [Sun Aug 28 00:42:52 2022] POST /identity/v3/services => generated 242 bytes in 212 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 28 00:42:53.677438 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dcd8318-921e-497e-b7cd-7f449fb862df None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:53.678250 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dcd8318-921e-497e-b7cd-7f449fb862df None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:53.678250 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dcd8318-921e-497e-b7cd-7f449fb862df None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:53.679668 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 504/1011] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:42:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 28 00:42:53.685474 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff6f8223-bafa-4adc-b1c0-d190ee357e27 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:53.686261 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff6f8223-bafa-4adc-b1c0-d190ee357e27 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:53.686261 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff6f8223-bafa-4adc-b1c0-d190ee357e27 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:53.705409 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-ff6f8223-bafa-4adc-b1c0-d190ee357e27 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:42:53.706618 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-ff6f8223-bafa-4adc-b1c0-d190ee357e27 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:42:53.714988 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:53.786791 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 508/1012] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:42:53 2022] POST /identity/v3/auth/tokens => generated 1813 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 28 00:42:54.047734 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dcd8318-921e-497e-b7cd-7f449fb862df None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:54.082171 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6083438-495e-4216-b132-086a5e250e69 None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:54.082449 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6083438-495e-4216-b132-086a5e250e69 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:54.082739 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6083438-495e-4216-b132-086a5e250e69 None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:54.084338 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 509/1013] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:42:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:42:54.089662 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5218b839-9925-47d6-af6c-fce356854abd None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:54.089940 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5218b839-9925-47d6-af6c-fce356854abd None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:54.090224 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5218b839-9925-47d6-af6c-fce356854abd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:54.108641 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-5218b839-9925-47d6-af6c-fce356854abd None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:42:54.110016 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-5218b839-9925-47d6-af6c-fce356854abd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:42:54.169137 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dcd8318-921e-497e-b7cd-7f449fb862df None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:54.186236 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 510/1014] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:42:54 2022] POST /identity/v3/auth/tokens => generated 1813 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 28 00:42:54.206608 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d07f6787-c14d-48df-823c-bd8e16b73d4f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:54.207253 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d07f6787-c14d-48df-823c-bd8e16b73d4f None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:54.207530 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d07f6787-c14d-48df-823c-bd8e16b73d4f None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:54.207802 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d07f6787-c14d-48df-823c-bd8e16b73d4f None admin] PATH_INFO: `/v3/domains/default` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:54.208954 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d07f6787-c14d-48df-823c-bd8e16b73d4f None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:54.210428 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d07f6787-c14d-48df-823c-bd8e16b73d4f None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:54.213006 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 505/1015] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:42:54 2022] GET /identity/v3/domains/default => generated 199 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:54.218380 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5218b839-9925-47d6-af6c-fce356854abd None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:54.287480 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:54.323240 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5218b839-9925-47d6-af6c-fce356854abd None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:54.358677 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92e3779a-dbb2-4192-a44e-fe56cab01639 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:54.359302 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92e3779a-dbb2-4192-a44e-fe56cab01639 None admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:54.359582 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92e3779a-dbb2-4192-a44e-fe56cab01639 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:54.359856 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92e3779a-dbb2-4192-a44e-fe56cab01639 None admin] PATH_INFO: `/v3/projects` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:54.360521 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92e3779a-dbb2-4192-a44e-fe56cab01639 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:54.361997 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92e3779a-dbb2-4192-a44e-fe56cab01639 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:54.401031 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 511/1016] 10.222.0.42 () {60 vars in 1141 bytes} [Sun Aug 28 00:42:54 2022] POST /identity/v3/projects => generated 311 bytes in 174 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:42:54.448345 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d07f6787-c14d-48df-823c-bd8e16b73d4f None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:54.520878 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:54.577945 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d07f6787-c14d-48df-823c-bd8e16b73d4f None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:54.617100 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5fbbfeb-5af1-4dbb-8ced-30de422d5856 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:54.617751 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5fbbfeb-5af1-4dbb-8ced-30de422d5856 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:54.618027 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5fbbfeb-5af1-4dbb-8ced-30de422d5856 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:54.618293 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5fbbfeb-5af1-4dbb-8ced-30de422d5856 None admin] PATH_INFO: `/v3/services/compute` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:54.618880 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5fbbfeb-5af1-4dbb-8ced-30de422d5856 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:54.620286 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5fbbfeb-5af1-4dbb-8ced-30de422d5856 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:54.635837 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-d5fbbfeb-5af1-4dbb-8ced-30de422d5856 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Aug 28 00:42:54.635837 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:54.635837 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 28 00:42:54.635837 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 28 00:42:54.635837 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 28 00:42:54.635837 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 28 00:42:54.635837 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 28 00:42:54.635837 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 28 00:42:54.635837 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Aug 28 00:42:54.635837 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:54.635837 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 28 00:42:54.635837 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:54.635837 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:54.635837 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:54.635837 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:54.635837 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:54.635837 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:54.636949 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:54.636949 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:54.636949 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:54.636949 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:54.636949 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:54.636949 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:54.636949 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 28 00:42:54.636949 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 28 00:42:54.636949 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 28 00:42:54.636949 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 28 00:42:54.636949 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:54.636949 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:54.636949 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:54.636949 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:54.636949 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:54.636949 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:54.637842 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:54.637842 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:54.637842 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:54.637842 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:54.637842 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:54.637842 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:54.637842 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:54.637842 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:54.637842 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:54.637842 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:54.637842 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 28 00:42:54.637842 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 28 00:42:54.637842 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Aug 28 00:42:54.637842 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:54.639448 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 506/1017] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Aug 28 00:42:54 2022] GET /identity/v3/services/compute => generated 88 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:54.645106 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92e3779a-dbb2-4192-a44e-fe56cab01639 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:54.816823 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92e3779a-dbb2-4192-a44e-fe56cab01639 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:54.853334 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:54.871116 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f707485c-9a5b-4acc-bff0-a9d7f8d723fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:54.871737 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f707485c-9a5b-4acc-bff0-a9d7f8d723fe None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:54.872046 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f707485c-9a5b-4acc-bff0-a9d7f8d723fe None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:54.872360 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f707485c-9a5b-4acc-bff0-a9d7f8d723fe None admin] PATH_INFO: `/v3/services` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:54.873143 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f707485c-9a5b-4acc-bff0-a9d7f8d723fe None admin] RBAC: Adding query filter params (name=compute) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:54.873453 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f707485c-9a5b-4acc-bff0-a9d7f8d723fe None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:54.874985 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f707485c-9a5b-4acc-bff0-a9d7f8d723fe None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:54.881428 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 512/1018] 10.222.0.42 () {56 vars in 1112 bytes} [Sun Aug 28 00:42:54 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 237 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:54.887205 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5fbbfeb-5af1-4dbb-8ced-30de422d5856 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:55.027613 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5fbbfeb-5af1-4dbb-8ced-30de422d5856 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:55.052051 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:55.074471 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbca2f7d-e9cb-4d6d-ae99-feac4c4b5ea2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:55.075106 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbca2f7d-e9cb-4d6d-ae99-feac4c4b5ea2 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:55.075405 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbca2f7d-e9cb-4d6d-ae99-feac4c4b5ea2 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:55.075716 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbca2f7d-e9cb-4d6d-ae99-feac4c4b5ea2 None admin] PATH_INFO: `/v3/services` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:55.076341 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbca2f7d-e9cb-4d6d-ae99-feac4c4b5ea2 None admin] RBAC: Adding query filter params (type=compute) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:55.076640 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbca2f7d-e9cb-4d6d-ae99-feac4c4b5ea2 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:55.078089 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbca2f7d-e9cb-4d6d-ae99-feac4c4b5ea2 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:55.084811 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 507/1019] 10.222.0.42 () {56 vars in 1112 bytes} [Sun Aug 28 00:42:54 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:55.091165 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f707485c-9a5b-4acc-bff0-a9d7f8d723fe None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:55.275241 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f707485c-9a5b-4acc-bff0-a9d7f8d723fe None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:55.313833 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3e5ca66-9da6-46ad-8e52-23d0cc48ac37 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:55.314454 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3e5ca66-9da6-46ad-8e52-23d0cc48ac37 None admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:55.314735 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3e5ca66-9da6-46ad-8e52-23d0cc48ac37 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:55.315005 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3e5ca66-9da6-46ad-8e52-23d0cc48ac37 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:55.315694 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3e5ca66-9da6-46ad-8e52-23d0cc48ac37 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:55.317123 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3e5ca66-9da6-46ad-8e52-23d0cc48ac37 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:55.345009 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 513/1020] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 00:42:55 2022] POST /identity/v3/endpoints => generated 334 bytes in 250 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:42:55.527271 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a43f56b7-2a5e-494a-a177-52456d8c091e None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:55.527560 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a43f56b7-2a5e-494a-a177-52456d8c091e None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:55.527882 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a43f56b7-2a5e-494a-a177-52456d8c091e None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:55.529309 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 508/1021] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:42:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 28 00:42:55.537450 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-878fc887-cb0e-4627-8941-632ac2b94f2d None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:55.537911 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-878fc887-cb0e-4627-8941-632ac2b94f2d None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:55.538368 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-878fc887-cb0e-4627-8941-632ac2b94f2d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:55.562134 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-878fc887-cb0e-4627-8941-632ac2b94f2d None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:42:55.563964 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-878fc887-cb0e-4627-8941-632ac2b94f2d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:42:55.611470 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:55.668906 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 514/1022] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:42:55 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 28 00:42:55.943135 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a43f56b7-2a5e-494a-a177-52456d8c091e None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:56.119852 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a43f56b7-2a5e-494a-a177-52456d8c091e None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:56.159190 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b97da09-16db-48d4-a162-06145fe602a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:56.159824 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b97da09-16db-48d4-a162-06145fe602a7 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:56.160114 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b97da09-16db-48d4-a162-06145fe602a7 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:56.160358 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b97da09-16db-48d4-a162-06145fe602a7 None admin] PATH_INFO: `/v3/services/image` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:56.160946 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b97da09-16db-48d4-a162-06145fe602a7 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:56.162400 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b97da09-16db-48d4-a162-06145fe602a7 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:56.171853 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-2b97da09-16db-48d4-a162-06145fe602a7 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Aug 28 00:42:56.171853 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:56.171853 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 28 00:42:56.171853 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 28 00:42:56.171853 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 28 00:42:56.171853 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 28 00:42:56.171853 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 28 00:42:56.171853 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 28 00:42:56.171853 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Aug 28 00:42:56.171853 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:56.171853 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 28 00:42:56.171853 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:56.171853 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:56.171853 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:56.171853 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:56.171853 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:56.171853 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:56.174243 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:56.174243 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:56.174243 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:56.174243 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:56.174243 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:56.174243 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:56.174243 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 28 00:42:56.174243 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 28 00:42:56.174243 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 28 00:42:56.174243 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 28 00:42:56.174243 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:56.174243 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:56.174243 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:56.174243 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:56.174243 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:56.174243 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:56.176057 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:56.176057 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:56.176057 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:56.176057 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:56.176057 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:56.176057 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:56.176057 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:56.176057 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:56.176057 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:56.176057 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:56.176057 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 28 00:42:56.176057 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 28 00:42:56.176057 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Aug 28 00:42:56.176057 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:56.179141 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 509/1023] 10.222.0.42 () {56 vars in 1105 bytes} [Sun Aug 28 00:42:55 2022] GET /identity/v3/services/image => generated 86 bytes in 236 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 28 00:42:56.181191 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-878fc887-cb0e-4627-8941-632ac2b94f2d None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:56.323742 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-878fc887-cb0e-4627-8941-632ac2b94f2d None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:56.362295 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-282e95de-7e67-4884-8aa9-c129575c5b6c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:56.362904 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-282e95de-7e67-4884-8aa9-c129575c5b6c None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:56.363173 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-282e95de-7e67-4884-8aa9-c129575c5b6c None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:56.363440 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-282e95de-7e67-4884-8aa9-c129575c5b6c None admin] PATH_INFO: `/v3/services` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:56.364082 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-282e95de-7e67-4884-8aa9-c129575c5b6c None admin] RBAC: Adding query filter params (name=image) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:56.364390 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-282e95de-7e67-4884-8aa9-c129575c5b6c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:56.365850 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-282e95de-7e67-4884-8aa9-c129575c5b6c None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:56.377024 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 515/1024] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:42:56 2022] GET /identity/v3/services?name=image => generated 122 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 28 00:42:56.381104 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b97da09-16db-48d4-a162-06145fe602a7 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:56.390563 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d811563c-e9ef-4388-8065-e8c5da7a4812 None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:56.390884 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d811563c-e9ef-4388-8065-e8c5da7a4812 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:56.391106 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d811563c-e9ef-4388-8065-e8c5da7a4812 None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:56.392587 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 516/1025] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:42:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 28 00:42:56.397928 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94ae5151-7f9c-4170-b2e4-b704224382f7 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:56.398217 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94ae5151-7f9c-4170-b2e4-b704224382f7 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:56.398485 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94ae5151-7f9c-4170-b2e4-b704224382f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:56.418658 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-94ae5151-7f9c-4170-b2e4-b704224382f7 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:42:56.420008 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-94ae5151-7f9c-4170-b2e4-b704224382f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:42:56.449866 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:56.457813 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:56.518832 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 517/1026] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:42:56 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 28 00:42:56.536708 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b97da09-16db-48d4-a162-06145fe602a7 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:56.581901 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9605b96-e425-4800-9edc-869f211bee33 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:56.582793 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9605b96-e425-4800-9edc-869f211bee33 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:56.583251 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9605b96-e425-4800-9edc-869f211bee33 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:56.583701 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9605b96-e425-4800-9edc-869f211bee33 None admin] PATH_INFO: `/v3/services` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:56.584373 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9605b96-e425-4800-9edc-869f211bee33 None admin] RBAC: Adding query filter params (type=image) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:56.584684 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9605b96-e425-4800-9edc-869f211bee33 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:56.586097 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9605b96-e425-4800-9edc-869f211bee33 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:56.591918 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 510/1027] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:42:56 2022] GET /identity/v3/services?type=image => generated 350 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:56.598159 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94ae5151-7f9c-4170-b2e4-b704224382f7 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:56.709223 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94ae5151-7f9c-4170-b2e4-b704224382f7 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:56.747539 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0f1b11c-00d0-4f19-bc7c-3e66c2b9dfc4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:56.748163 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0f1b11c-00d0-4f19-bc7c-3e66c2b9dfc4 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:56.748464 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0f1b11c-00d0-4f19-bc7c-3e66c2b9dfc4 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:56.748743 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0f1b11c-00d0-4f19-bc7c-3e66c2b9dfc4 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:56.749564 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0f1b11c-00d0-4f19-bc7c-3e66c2b9dfc4 None admin] RBAC: Adding query filter params (interface=public, service_id=5dd704a804e74838986ead5c12603a2d, region_id=RegionOne) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:56.749813 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0f1b11c-00d0-4f19-bc7c-3e66c2b9dfc4 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:56.751224 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0f1b11c-00d0-4f19-bc7c-3e66c2b9dfc4 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:56.758824 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 518/1028] 10.222.0.42 () {56 vars in 1251 bytes} [Sun Aug 28 00:42:56 2022] GET /identity/v3/endpoints?service_id=5dd704a804e74838986ead5c12603a2d&interface=public®ion_id=RegionOne => generated 194 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 28 00:42:56.765475 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9605b96-e425-4800-9edc-869f211bee33 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:56.785705 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0f1b11c-00d0-4f19-bc7c-3e66c2b9dfc4 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:56.814721 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:56.923510 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9605b96-e425-4800-9edc-869f211bee33 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:56.923760 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0f1b11c-00d0-4f19-bc7c-3e66c2b9dfc4 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:56.937216 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:56.964858 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebc5457a-d5ee-4419-a5b7-00ea63a93e5d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:56.965642 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebc5457a-d5ee-4419-a5b7-00ea63a93e5d None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:56.965939 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebc5457a-d5ee-4419-a5b7-00ea63a93e5d None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:56.966265 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebc5457a-d5ee-4419-a5b7-00ea63a93e5d None admin] PATH_INFO: `/v3/domains/default` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:56.967601 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebc5457a-d5ee-4419-a5b7-00ea63a93e5d None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:56.969250 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebc5457a-d5ee-4419-a5b7-00ea63a93e5d None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:56.971127 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 519/1029] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:42:56 2022] GET /identity/v3/domains/default => generated 199 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 28 00:42:56.972714 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81bab892-0abe-4269-97f1-c0ad7f9f8499 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:56.973478 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81bab892-0abe-4269-97f1-c0ad7f9f8499 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:56.973630 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81bab892-0abe-4269-97f1-c0ad7f9f8499 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:56.973922 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81bab892-0abe-4269-97f1-c0ad7f9f8499 None admin] PATH_INFO: `/v3/services` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:56.974484 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81bab892-0abe-4269-97f1-c0ad7f9f8499 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:56.975849 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81bab892-0abe-4269-97f1-c0ad7f9f8499 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:56.976611 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebc5457a-d5ee-4419-a5b7-00ea63a93e5d None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:56.985172 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 511/1030] 10.222.0.42 () {56 vars in 1087 bytes} [Sun Aug 28 00:42:56 2022] GET /identity/v3/services => generated 1957 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 28 00:42:57.130520 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:57.144300 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebc5457a-d5ee-4419-a5b7-00ea63a93e5d None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:57.188124 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be771ce6-167c-474e-8c44-a54dc4601721 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:57.189156 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be771ce6-167c-474e-8c44-a54dc4601721 None admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:57.189617 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be771ce6-167c-474e-8c44-a54dc4601721 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:57.190073 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be771ce6-167c-474e-8c44-a54dc4601721 None admin] PATH_INFO: `/v3/users` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:57.191199 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be771ce6-167c-474e-8c44-a54dc4601721 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:57.193792 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be771ce6-167c-474e-8c44-a54dc4601721 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:57.231605 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 520/1031] 10.222.0.42 () {60 vars in 1132 bytes} [Sun Aug 28 00:42:56 2022] POST /identity/v3/users => generated 285 bytes in 254 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 28 00:42:57.500293 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9c3bbc0-3e34-4f30-8aed-c5a6b26e7c46 None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:57.500707 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9c3bbc0-3e34-4f30-8aed-c5a6b26e7c46 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:57.501252 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9c3bbc0-3e34-4f30-8aed-c5a6b26e7c46 None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:57.504007 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 512/1032] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:42:57 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:42:57.511800 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2653459c-caf2-4d19-8172-69e7863e43d7 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:57.512088 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2653459c-caf2-4d19-8172-69e7863e43d7 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:57.512554 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2653459c-caf2-4d19-8172-69e7863e43d7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:57.532176 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-2653459c-caf2-4d19-8172-69e7863e43d7 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:42:57.533447 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-2653459c-caf2-4d19-8172-69e7863e43d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:42:57.604662 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 521/1033] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:42:57 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 28 00:42:57.867790 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9c3bbc0-3e34-4f30-8aed-c5a6b26e7c46 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:57.978808 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9c3bbc0-3e34-4f30-8aed-c5a6b26e7c46 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:58.017717 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6c37296-3591-431a-b29c-3516832ad52e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:58.018380 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6c37296-3591-431a-b29c-3516832ad52e None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:58.018655 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6c37296-3591-431a-b29c-3516832ad52e None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:58.018957 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6c37296-3591-431a-b29c-3516832ad52e None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:58.030324 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6c37296-3591-431a-b29c-3516832ad52e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 28 00:42:58.035195 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-f6c37296-3591-431a-b29c-3516832ad52e None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 28 00:42:58.035195 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:58.035195 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:58.035195 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:58.035195 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:58.035195 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:58.035195 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:58.035195 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:58.035195 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:58.035195 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:58.035195 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:58.035195 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:58.035195 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 28 00:42:58.035195 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 28 00:42:58.035195 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 28 00:42:58.035195 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 28 00:42:58.037050 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 28 00:42:58.037050 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 28 00:42:58.037050 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 28 00:42:58.037050 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 28 00:42:58.037050 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:58.037050 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:58.037050 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:58.037050 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:58.037050 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:58.037050 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:58.037050 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:58.037050 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:58.037050 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:58.037050 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:58.037050 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:58.037050 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:58.038757 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:58.038757 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:58.038757 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:58.038757 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:58.038757 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 28 00:42:58.038757 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 28 00:42:58.038757 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 28 00:42:58.038757 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 28 00:42:58.038757 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 28 00:42:58.038757 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 28 00:42:58.038757 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 28 00:42:58.038757 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:58.041321 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 513/1034] 10.222.0.42 () {56 vars in 1120 bytes} [Sun Aug 28 00:42:57 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:58.043481 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2653459c-caf2-4d19-8172-69e7863e43d7 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:58.147455 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:58.189495 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2653459c-caf2-4d19-8172-69e7863e43d7 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:58.250299 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5ddd314-3871-4870-9d5b-1d2fd6452387 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:58.251324 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5ddd314-3871-4870-9d5b-1d2fd6452387 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:58.251814 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5ddd314-3871-4870-9d5b-1d2fd6452387 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:58.252293 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5ddd314-3871-4870-9d5b-1d2fd6452387 None admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:58.253545 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5ddd314-3871-4870-9d5b-1d2fd6452387 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:58.254088 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5ddd314-3871-4870-9d5b-1d2fd6452387 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:58.256630 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5ddd314-3871-4870-9d5b-1d2fd6452387 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:58.272704 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 522/1035] 10.222.0.42 () {56 vars in 1115 bytes} [Sun Aug 28 00:42:58 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 230 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:58.278926 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6c37296-3591-431a-b29c-3516832ad52e None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:58.340712 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:58.431213 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6c37296-3591-431a-b29c-3516832ad52e None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:58.468573 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f95fefe9-7ab2-4aed-8fa8-0b60226a5487 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:58.469226 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f95fefe9-7ab2-4aed-8fa8-0b60226a5487 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:58.469492 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f95fefe9-7ab2-4aed-8fa8-0b60226a5487 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:58.469763 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f95fefe9-7ab2-4aed-8fa8-0b60226a5487 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:58.480606 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f95fefe9-7ab2-4aed-8fa8-0b60226a5487 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:58.482075 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f95fefe9-7ab2-4aed-8fa8-0b60226a5487 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:58.499230 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-f95fefe9-7ab2-4aed-8fa8-0b60226a5487 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:42:58.499230 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:58.499230 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:58.499230 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:58.499230 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:58.499230 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:58.499230 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:58.499230 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:58.499230 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:58.499230 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:58.499230 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:58.499230 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:58.499230 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 28 00:42:58.499230 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 28 00:42:58.499230 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 28 00:42:58.499230 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 28 00:42:58.501514 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:58.501514 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:58.501514 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:58.501514 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:58.501514 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:58.501514 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:58.501514 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:58.501514 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:58.501514 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:58.501514 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:58.501514 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:58.501514 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:58.501514 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:58.501514 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:58.501514 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:58.501514 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:58.501514 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 28 00:42:58.503233 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 28 00:42:58.503233 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:42:58.503233 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:58.505083 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 514/1036] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:42:58 2022] GET /identity/v3/domains/Default => generated 87 bytes in 223 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:58.507227 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5ddd314-3871-4870-9d5b-1d2fd6452387 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:58.670502 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5ddd314-3871-4870-9d5b-1d2fd6452387 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:58.728696 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2edfd37b-2ef9-4666-8017-f6f570a6a765 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:58.729542 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2edfd37b-2ef9-4666-8017-f6f570a6a765 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:58.729984 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2edfd37b-2ef9-4666-8017-f6f570a6a765 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:58.730563 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2edfd37b-2ef9-4666-8017-f6f570a6a765 None admin] PATH_INFO: `/v3/domains` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:58.731749 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2edfd37b-2ef9-4666-8017-f6f570a6a765 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:58.732337 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2edfd37b-2ef9-4666-8017-f6f570a6a765 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:58.734988 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2edfd37b-2ef9-4666-8017-f6f570a6a765 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:58.749445 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:58.759985 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 523/1037] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Aug 28 00:42:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 250 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:42:58.762303 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f95fefe9-7ab2-4aed-8fa8-0b60226a5487 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:58.881927 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:58.905197 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f95fefe9-7ab2-4aed-8fa8-0b60226a5487 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:58.945608 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0b9d367-03c8-40aa-b130-8ceb171c2490 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:58.946250 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0b9d367-03c8-40aa-b130-8ceb171c2490 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:58.946520 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0b9d367-03c8-40aa-b130-8ceb171c2490 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:58.946785 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0b9d367-03c8-40aa-b130-8ceb171c2490 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:58.956064 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0b9d367-03c8-40aa-b130-8ceb171c2490 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:58.957603 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0b9d367-03c8-40aa-b130-8ceb171c2490 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:58.970423 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-a0b9d367-03c8-40aa-b130-8ceb171c2490 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 28 00:42:58.970423 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:58.970423 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:58.970423 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:58.970423 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:58.970423 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:58.970423 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:58.970423 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:58.970423 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:58.970423 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:58.970423 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:58.970423 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:58.970423 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 28 00:42:58.970423 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 28 00:42:58.970423 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 28 00:42:58.970423 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 28 00:42:58.972383 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:58.972383 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:58.972383 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 28 00:42:58.972383 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 28 00:42:58.972383 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 28 00:42:58.972383 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 28 00:42:58.972383 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:58.972383 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:58.972383 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:58.972383 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:58.972383 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:58.972383 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:58.972383 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:58.972383 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:58.972383 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:58.972383 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:58.972383 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:58.972383 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:58.974389 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:58.974389 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:58.974389 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 28 00:42:58.974389 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 28 00:42:58.974389 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 28 00:42:58.974389 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 28 00:42:58.974389 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 28 00:42:58.974389 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 28 00:42:58.974389 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 28 00:42:58.974389 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:58.976896 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 515/1038] 10.222.0.42 () {56 vars in 1093 bytes} [Sun Aug 28 00:42:58 2022] GET /identity/v3/users/nova => generated 82 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:58.979000 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2edfd37b-2ef9-4666-8017-f6f570a6a765 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:59.123832 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb649cf3-6e67-496f-b47d-4821f965f536 None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:59.124542 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb649cf3-6e67-496f-b47d-4821f965f536 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:59.124542 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb649cf3-6e67-496f-b47d-4821f965f536 None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:59.125974 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 516/1039] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:42:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 28 00:42:59.131283 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9b41835-ffcc-4a06-bfe6-3baf1333a505 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:59.131572 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9b41835-ffcc-4a06-bfe6-3baf1333a505 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:59.131848 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9b41835-ffcc-4a06-bfe6-3baf1333a505 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:59.151921 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-f9b41835-ffcc-4a06-bfe6-3baf1333a505 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:42:59.153182 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-f9b41835-ffcc-4a06-bfe6-3baf1333a505 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:42:59.154174 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2edfd37b-2ef9-4666-8017-f6f570a6a765 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:59.195932 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c01af510-1a4a-4032-b7e5-024ec95f4fa1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:59.196946 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c01af510-1a4a-4032-b7e5-024ec95f4fa1 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:59.196946 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c01af510-1a4a-4032-b7e5-024ec95f4fa1 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:59.197298 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c01af510-1a4a-4032-b7e5-024ec95f4fa1 None admin] PATH_INFO: `/v3/users` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:59.197908 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c01af510-1a4a-4032-b7e5-024ec95f4fa1 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:42:59.198180 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c01af510-1a4a-4032-b7e5-024ec95f4fa1 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:59.199637 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c01af510-1a4a-4032-b7e5-024ec95f4fa1 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:59.219286 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 524/1040] 10.222.0.42 () {56 vars in 1133 bytes} [Sun Aug 28 00:42:58 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 242 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 28 00:42:59.220663 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79c75c7a-b55b-4073-b360-da13e92daea2 None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:59.220993 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79c75c7a-b55b-4073-b360-da13e92daea2 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:59.221289 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79c75c7a-b55b-4073-b360-da13e92daea2 None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:59.222495 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 525/1041] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:42:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:42:59.226002 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79c75c7a-b55b-4073-b360-da13e92daea2 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:59.235115 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 517/1042] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:42:59 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 28 00:42:59.236440 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-326610de-e905-4263-ab6c-308d6226496a None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:59.236732 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-326610de-e905-4263-ab6c-308d6226496a None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:59.237037 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-326610de-e905-4263-ab6c-308d6226496a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:59.258094 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-326610de-e905-4263-ab6c-308d6226496a None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:42:59.259291 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-326610de-e905-4263-ab6c-308d6226496a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:42:59.300486 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:59.343873 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 518/1043] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:42:59 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 28 00:42:59.372098 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79c75c7a-b55b-4073-b360-da13e92daea2 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:59.431169 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfcb6a8f-b31b-4b9e-a6de-d5b2b5713a9e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:59.432201 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfcb6a8f-b31b-4b9e-a6de-d5b2b5713a9e None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:59.432724 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfcb6a8f-b31b-4b9e-a6de-d5b2b5713a9e None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:59.433223 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfcb6a8f-b31b-4b9e-a6de-d5b2b5713a9e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:59.502477 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-326610de-e905-4263-ab6c-308d6226496a None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:59.512054 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:59.548576 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfcb6a8f-b31b-4b9e-a6de-d5b2b5713a9e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:59.550118 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfcb6a8f-b31b-4b9e-a6de-d5b2b5713a9e None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:59.613511 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:59.624738 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-326610de-e905-4263-ab6c-308d6226496a None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:59.658810 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 526/1044] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Aug 28 00:42:59 2022] GET /identity/v3/auth/tokens => generated 1969 bytes in 433 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Aug 28 00:42:59.659916 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfcb6a8f-b31b-4b9e-a6de-d5b2b5713a9e None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:59.667397 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f254d32-ad1a-4636-a7c6-f395057752b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:59.668031 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f254d32-ad1a-4636-a7c6-f395057752b8 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:59.668279 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f254d32-ad1a-4636-a7c6-f395057752b8 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:59.668572 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f254d32-ad1a-4636-a7c6-f395057752b8 None admin] PATH_INFO: `/v3/services/image` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:59.669153 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f254d32-ad1a-4636-a7c6-f395057752b8 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:59.670708 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f254d32-ad1a-4636-a7c6-f395057752b8 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:59.680574 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-1f254d32-ad1a-4636-a7c6-f395057752b8 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Aug 28 00:42:59.680574 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:59.680574 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 28 00:42:59.680574 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 28 00:42:59.680574 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 28 00:42:59.680574 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 28 00:42:59.680574 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 28 00:42:59.680574 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 28 00:42:59.680574 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Aug 28 00:42:59.680574 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:59.680574 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 28 00:42:59.680574 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:59.680574 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:59.680574 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:59.680574 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:59.680574 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:59.680574 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:59.681665 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:59.681665 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:59.681665 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:59.681665 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:59.681665 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:59.681665 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:59.681665 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 28 00:42:59.681665 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 28 00:42:59.681665 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 28 00:42:59.681665 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 28 00:42:59.681665 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:59.681665 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:59.681665 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:59.681665 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:59.681665 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:59.681665 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:59.682513 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:59.682513 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:59.682513 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:59.682513 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:59.682513 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:59.682513 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:59.682513 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:59.682513 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:59.682513 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:59.682513 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:59.682513 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 28 00:42:59.682513 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 28 00:42:59.682513 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Aug 28 00:42:59.682513 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:59.682513 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 519/1045] 10.222.0.42 () {56 vars in 1105 bytes} [Sun Aug 28 00:42:59 2022] GET /identity/v3/services/image => generated 86 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 28 00:42:59.683339 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f254d32-ad1a-4636-a7c6-f395057752b8 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:59.754513 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:59.767983 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfcb6a8f-b31b-4b9e-a6de-d5b2b5713a9e None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:59.786733 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f254d32-ad1a-4636-a7c6-f395057752b8 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:59.802190 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-459b442e-9ea7-454d-9f19-dd952e219899 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:59.802821 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-459b442e-9ea7-454d-9f19-dd952e219899 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:59.803101 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-459b442e-9ea7-454d-9f19-dd952e219899 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:59.803364 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-459b442e-9ea7-454d-9f19-dd952e219899 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:59.813728 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-459b442e-9ea7-454d-9f19-dd952e219899 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 28 00:42:59.818553 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-459b442e-9ea7-454d-9f19-dd952e219899 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 28 00:42:59.818553 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:59.818553 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:59.818553 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:59.818553 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:59.818553 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:59.818553 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:59.818553 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:59.818553 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:59.818553 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:59.818553 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:59.818553 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:59.818553 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 28 00:42:59.818553 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 28 00:42:59.818553 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 28 00:42:59.818553 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 28 00:42:59.819904 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 28 00:42:59.819904 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 28 00:42:59.819904 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 28 00:42:59.819904 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 28 00:42:59.819904 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:59.819904 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:59.819904 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:59.819904 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:59.819904 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:59.819904 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:59.819904 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:59.819904 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:59.819904 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:59.819904 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:59.819904 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:59.819904 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:59.822237 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:59.822237 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:59.822237 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:59.822237 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:59.822237 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 28 00:42:59.822237 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 28 00:42:59.822237 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 28 00:42:59.822237 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 28 00:42:59.822237 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 28 00:42:59.822237 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 28 00:42:59.822237 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 28 00:42:59.822237 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:42:59.822237 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 527/1046] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Aug 28 00:42:59 2022] GET /identity/v3/roles/admin => generated 83 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:59.822237 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-459b442e-9ea7-454d-9f19-dd952e219899 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:59.826005 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b59f813c-39b9-490d-bc85-0a52bca70cfb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:42:59.826660 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b59f813c-39b9-490d-bc85-0a52bca70cfb None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:42:59.826924 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b59f813c-39b9-490d-bc85-0a52bca70cfb None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:42:59.827194 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b59f813c-39b9-490d-bc85-0a52bca70cfb None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:42:59.841360 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b59f813c-39b9-490d-bc85-0a52bca70cfb None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:42:59.842839 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b59f813c-39b9-490d-bc85-0a52bca70cfb None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:42:59.853548 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:42:59.858277 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-b59f813c-39b9-490d-bc85-0a52bca70cfb None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:42:59.858277 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:42:59.858277 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:42:59.858277 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:42:59.858277 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:42:59.858277 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:42:59.858277 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:42:59.858277 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:42:59.858277 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:42:59.858277 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:42:59.858277 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:42:59.858277 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:42:59.858277 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 28 00:42:59.858277 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 28 00:42:59.858277 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 28 00:42:59.858277 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 28 00:42:59.859836 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:42:59.859836 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:42:59.859836 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:42:59.859836 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:42:59.859836 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:42:59.859836 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:42:59.859836 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:42:59.859836 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:42:59.859836 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:42:59.859836 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:42:59.859836 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:42:59.859836 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:42:59.859836 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:42:59.859836 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:42:59.859836 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:42:59.859836 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:42:59.859836 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 28 00:42:59.860935 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 28 00:42:59.860935 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:42:59.860935 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:42:59.860935 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 520/1047] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:42:59 2022] GET /identity/v3/domains/Default => generated 87 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:42:59.860935 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b59f813c-39b9-490d-bc85-0a52bca70cfb None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:42:59.954710 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-459b442e-9ea7-454d-9f19-dd952e219899 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:59.974561 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b59f813c-39b9-490d-bc85-0a52bca70cfb None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:42:59.999463 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d725c875-a7f0-4baa-a665-11595f4723f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:00.000069 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d725c875-a7f0-4baa-a665-11595f4723f6 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:00.000347 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d725c875-a7f0-4baa-a665-11595f4723f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:00.000601 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d725c875-a7f0-4baa-a665-11595f4723f6 None admin] PATH_INFO: `/v3/services` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:00.001257 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d725c875-a7f0-4baa-a665-11595f4723f6 None admin] RBAC: Adding query filter params (name=image) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:43:00.001554 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d725c875-a7f0-4baa-a665-11595f4723f6 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:43:00.002960 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d725c875-a7f0-4baa-a665-11595f4723f6 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:43:00.003082 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:43:00.008841 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 528/1048] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:42:59 2022] GET /identity/v3/services?name=image => generated 122 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:43:00.009653 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d725c875-a7f0-4baa-a665-11595f4723f6 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:00.021290 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d9e5349-953f-41f3-8029-7e5e8d20657e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:00.022202 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d9e5349-953f-41f3-8029-7e5e8d20657e None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:00.022570 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d9e5349-953f-41f3-8029-7e5e8d20657e None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:00.023007 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d9e5349-953f-41f3-8029-7e5e8d20657e None admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:00.023862 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d9e5349-953f-41f3-8029-7e5e8d20657e None admin] RBAC: Adding query filter params (name=admin) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:43:00.024251 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d9e5349-953f-41f3-8029-7e5e8d20657e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:43:00.026030 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d9e5349-953f-41f3-8029-7e5e8d20657e None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:43:00.042852 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 521/1049] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Aug 28 00:42:59 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:43:00.042852 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d9e5349-953f-41f3-8029-7e5e8d20657e None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:00.101737 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:43:00.126468 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d725c875-a7f0-4baa-a665-11595f4723f6 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:00.159668 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d9e5349-953f-41f3-8029-7e5e8d20657e None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:00.182611 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ded2a56f-793f-48d2-8f67-ec82d896cc8a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:00.183660 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ded2a56f-793f-48d2-8f67-ec82d896cc8a None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:00.184100 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ded2a56f-793f-48d2-8f67-ec82d896cc8a None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:00.184530 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ded2a56f-793f-48d2-8f67-ec82d896cc8a None admin] PATH_INFO: `/v3/domains` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:00.185673 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ded2a56f-793f-48d2-8f67-ec82d896cc8a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:43:00.186148 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ded2a56f-793f-48d2-8f67-ec82d896cc8a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:43:00.188503 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ded2a56f-793f-48d2-8f67-ec82d896cc8a None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:43:00.198345 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e85b9e4-e6cb-4877-83e6-6473ebfa5ce8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:00.198971 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e85b9e4-e6cb-4877-83e6-6473ebfa5ce8 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:00.199250 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e85b9e4-e6cb-4877-83e6-6473ebfa5ce8 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:00.199520 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e85b9e4-e6cb-4877-83e6-6473ebfa5ce8 None admin] PATH_INFO: `/v3/services` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:00.200193 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e85b9e4-e6cb-4877-83e6-6473ebfa5ce8 None admin] RBAC: Adding query filter params (type=image) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:43:00.200541 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e85b9e4-e6cb-4877-83e6-6473ebfa5ce8 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:43:00.201991 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e85b9e4-e6cb-4877-83e6-6473ebfa5ce8 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:43:00.207855 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 522/1050] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:43:00 2022] GET /identity/v3/services?type=image => generated 350 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:43:00.212865 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 529/1051] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Aug 28 00:43:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:43:00.213687 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e85b9e4-e6cb-4877-83e6-6473ebfa5ce8 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:00.214240 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ded2a56f-793f-48d2-8f67-ec82d896cc8a None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:00.273534 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:43:00.335537 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e85b9e4-e6cb-4877-83e6-6473ebfa5ce8 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:00.337975 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ded2a56f-793f-48d2-8f67-ec82d896cc8a None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:00.371505 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d666dda-9219-42d4-8637-f1b271b89c77 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:00.372117 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d666dda-9219-42d4-8637-f1b271b89c77 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:00.372486 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d666dda-9219-42d4-8637-f1b271b89c77 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:00.372758 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d666dda-9219-42d4-8637-f1b271b89c77 None admin] PATH_INFO: `/v3/users/d95232208be944a6ae3ffbaf7007693d` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:00.374497 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c47bdda-00a2-4809-803b-78f7618433d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:00.375236 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c47bdda-00a2-4809-803b-78f7618433d4 None admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:00.375606 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c47bdda-00a2-4809-803b-78f7618433d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:00.375960 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c47bdda-00a2-4809-803b-78f7618433d4 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:00.376769 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c47bdda-00a2-4809-803b-78f7618433d4 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:43:00.378276 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c47bdda-00a2-4809-803b-78f7618433d4 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:43:00.388143 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d666dda-9219-42d4-8637-f1b271b89c77 None admin] RBAC: Authorizing `identity:get_user(user_id=d95232208be944a6ae3ffbaf7007693d)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:43:00.389644 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d666dda-9219-42d4-8637-f1b271b89c77 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:43:00.391512 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 523/1052] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Aug 28 00:43:00 2022] GET /identity/v3/users/d95232208be944a6ae3ffbaf7007693d => generated 285 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:43:00.392363 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d666dda-9219-42d4-8637-f1b271b89c77 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:00.395597 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:43:00.409053 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 530/1053] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 00:43:00 2022] POST /identity/v3/endpoints => generated 327 bytes in 191 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:43:00.410197 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c47bdda-00a2-4809-803b-78f7618433d4 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:00.542620 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c47bdda-00a2-4809-803b-78f7618433d4 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:00.553283 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d666dda-9219-42d4-8637-f1b271b89c77 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:00.566610 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:43:00.580846 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83ce02a2-4443-40a2-8b34-9095ea78b387 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:00.581498 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83ce02a2-4443-40a2-8b34-9095ea78b387 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:00.581787 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83ce02a2-4443-40a2-8b34-9095ea78b387 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:00.582106 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83ce02a2-4443-40a2-8b34-9095ea78b387 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:00.613450 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31172745-18ba-4bcf-883d-3c9a04af93c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:00.614767 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31172745-18ba-4bcf-883d-3c9a04af93c7 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:00.615151 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31172745-18ba-4bcf-883d-3c9a04af93c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:00.615593 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31172745-18ba-4bcf-883d-3c9a04af93c7 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:00.626975 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31172745-18ba-4bcf-883d-3c9a04af93c7 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:43:00.629104 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31172745-18ba-4bcf-883d-3c9a04af93c7 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:43:00.648124 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-31172745-18ba-4bcf-883d-3c9a04af93c7 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 28 00:43:00.648124 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:43:00.648124 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:43:00.648124 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:43:00.648124 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:43:00.648124 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:43:00.648124 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:43:00.648124 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:43:00.648124 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:43:00.648124 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:43:00.648124 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:43:00.648124 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:43:00.648124 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 28 00:43:00.648124 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 28 00:43:00.648124 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 28 00:43:00.649270 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 28 00:43:00.649270 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:43:00.649270 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:43:00.649270 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:43:00.649270 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:43:00.649270 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:43:00.649270 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:43:00.649270 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:43:00.649270 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:43:00.649270 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:43:00.649270 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:43:00.649270 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:43:00.649270 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:43:00.649270 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:43:00.649270 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:43:00.649270 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:43:00.649270 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:43:00.650184 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 28 00:43:00.650184 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 28 00:43:00.650184 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 28 00:43:00.650184 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 28 00:43:00.650184 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 28 00:43:00.650184 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 28 00:43:00.650184 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 28 00:43:00.650184 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:43:00.650184 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 524/1054] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Aug 28 00:43:00 2022] GET /identity/v3/projects/service => generated 88 bytes in 258 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:43:00.657241 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31172745-18ba-4bcf-883d-3c9a04af93c7 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:00.665674 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:43:00.678589 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83ce02a2-4443-40a2-8b34-9095ea78b387 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:43:00.681365 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83ce02a2-4443-40a2-8b34-9095ea78b387 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:43:00.783314 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31172745-18ba-4bcf-883d-3c9a04af93c7 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:00.810919 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 531/1055] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Aug 28 00:43:00 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 401 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 28 00:43:00.817347 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83ce02a2-4443-40a2-8b34-9095ea78b387 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:00.838395 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa9b28cf-28a5-484e-ac5a-20c666c57da2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:00.839269 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa9b28cf-28a5-484e-ac5a-20c666c57da2 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:00.839707 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa9b28cf-28a5-484e-ac5a-20c666c57da2 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:00.840143 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa9b28cf-28a5-484e-ac5a-20c666c57da2 None admin] PATH_INFO: `/v3/projects` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:00.841404 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa9b28cf-28a5-484e-ac5a-20c666c57da2 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:43:00.841781 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa9b28cf-28a5-484e-ac5a-20c666c57da2 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:43:00.844068 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa9b28cf-28a5-484e-ac5a-20c666c57da2 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:43:00.869005 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 525/1056] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Aug 28 00:43:00 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 28 00:43:00.870876 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa9b28cf-28a5-484e-ac5a-20c666c57da2 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:00.913581 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:43:00.980580 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:43:00.990859 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83ce02a2-4443-40a2-8b34-9095ea78b387 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:01.029034 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa9b28cf-28a5-484e-ac5a-20c666c57da2 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:01.046441 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e2c7232-a457-475e-a7fb-318a207bee2f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:01.047257 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e2c7232-a457-475e-a7fb-318a207bee2f None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:01.047617 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e2c7232-a457-475e-a7fb-318a207bee2f None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:01.048013 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e2c7232-a457-475e-a7fb-318a207bee2f None admin] PATH_INFO: `/v3/projects/a661c03290754db3864357aaaffd5aa3` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:01.049560 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e2c7232-a457-475e-a7fb-318a207bee2f None admin] RBAC: Authorizing `identity:get_project(project_id=a661c03290754db3864357aaaffd5aa3)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:43:01.051267 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e2c7232-a457-475e-a7fb-318a207bee2f None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:43:01.053675 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 532/1057] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Aug 28 00:43:00 2022] GET /identity/v3/projects/a661c03290754db3864357aaaffd5aa3 => generated 311 bytes in 237 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 28 00:43:01.060298 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e2c7232-a457-475e-a7fb-318a207bee2f None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:01.073685 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23fe247a-7bb8-4a3f-bae4-17cac52bb7f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:01.074192 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23fe247a-7bb8-4a3f-bae4-17cac52bb7f3 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:01.074465 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23fe247a-7bb8-4a3f-bae4-17cac52bb7f3 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:01.074736 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23fe247a-7bb8-4a3f-bae4-17cac52bb7f3 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:01.075491 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23fe247a-7bb8-4a3f-bae4-17cac52bb7f3 None admin] RBAC: Adding query filter params (role.id=2c6107d249594bd3abf791034b71e98f, scope.project.id=f53e6fdb369b480fbebb4086a505df67, user.id=168b07df50c24593ba73dbde5b037aed) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:43:01.075840 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23fe247a-7bb8-4a3f-bae4-17cac52bb7f3 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:43:01.077396 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23fe247a-7bb8-4a3f-bae4-17cac52bb7f3 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:43:01.089004 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 526/1058] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Aug 28 00:43:00 2022] GET /identity/v3/role_assignments?user.id=168b07df50c24593ba73dbde5b037aed&scope.project.id=f53e6fdb369b480fbebb4086a505df67&role.id=2c6107d249594bd3abf791034b71e98f => generated 259 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 28 00:43:01.193466 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e2c7232-a457-475e-a7fb-318a207bee2f None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:01.234554 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0ef8851-5690-48d2-824f-d992fbeb1a4d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:01.235263 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0ef8851-5690-48d2-824f-d992fbeb1a4d None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:01.235497 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0ef8851-5690-48d2-824f-d992fbeb1a4d None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:01.235699 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0ef8851-5690-48d2-824f-d992fbeb1a4d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:01.236474 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0ef8851-5690-48d2-824f-d992fbeb1a4d None admin] RBAC: Adding query filter params (role.id=f08d6043261747e3b4763c744eef7dfb, scope.project.id=a661c03290754db3864357aaaffd5aa3, user.id=d95232208be944a6ae3ffbaf7007693d) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:43:01.236777 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0ef8851-5690-48d2-824f-d992fbeb1a4d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:43:01.238278 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0ef8851-5690-48d2-824f-d992fbeb1a4d None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:43:01.245101 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 533/1059] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Aug 28 00:43:01 2022] GET /identity/v3/role_assignments?user.id=d95232208be944a6ae3ffbaf7007693d&scope.project.id=a661c03290754db3864357aaaffd5aa3&role.id=f08d6043261747e3b4763c744eef7dfb => generated 259 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:43:02.458292 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-618befb1-c001-404e-8512-5405ef784942 None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:02.458824 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-618befb1-c001-404e-8512-5405ef784942 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:02.459220 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-618befb1-c001-404e-8512-5405ef784942 None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:02.461699 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 527/1060] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:43:02 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:43:02.467525 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-459cfc89-dc31-47be-93e3-52514928767e None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:02.468287 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-459cfc89-dc31-47be-93e3-52514928767e None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:02.468287 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-459cfc89-dc31-47be-93e3-52514928767e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:02.487363 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-459cfc89-dc31-47be-93e3-52514928767e None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:43:02.488772 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-459cfc89-dc31-47be-93e3-52514928767e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:43:02.511878 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:43:02.562763 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 534/1061] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:43:02 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 28 00:43:02.827061 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-618befb1-c001-404e-8512-5405ef784942 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:02.986642 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:43:03.006594 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-618befb1-c001-404e-8512-5405ef784942 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:03.069639 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b5b77b7-5c1c-4ceb-85fa-b82e69adf24b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:03.070695 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b5b77b7-5c1c-4ceb-85fa-b82e69adf24b None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:03.071194 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b5b77b7-5c1c-4ceb-85fa-b82e69adf24b None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:03.071689 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b5b77b7-5c1c-4ceb-85fa-b82e69adf24b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:03.090646 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3944e0d-fff4-42b1-a33d-c35f633eda49 None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:03.090862 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3944e0d-fff4-42b1-a33d-c35f633eda49 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:03.091171 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3944e0d-fff4-42b1-a33d-c35f633eda49 None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:03.092710 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 535/1062] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:43:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:43:03.098059 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-405989cf-d8a0-46c9-9527-6c200d27f5cb None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:03.098342 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-405989cf-d8a0-46c9-9527-6c200d27f5cb None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:03.098602 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-405989cf-d8a0-46c9-9527-6c200d27f5cb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:03.117606 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-405989cf-d8a0-46c9-9527-6c200d27f5cb None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:43:03.118816 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-405989cf-d8a0-46c9-9527-6c200d27f5cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:43:03.191758 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b5b77b7-5c1c-4ceb-85fa-b82e69adf24b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:43:03.194054 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b5b77b7-5c1c-4ceb-85fa-b82e69adf24b None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:43:03.208463 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 536/1063] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:43:03 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 28 00:43:03.210682 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b633d71-01c9-4bae-b861-49bb547c4ef9 None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:03.211131 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b633d71-01c9-4bae-b861-49bb547c4ef9 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:03.211569 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b633d71-01c9-4bae-b861-49bb547c4ef9 None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:03.213911 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 537/1064] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:43:03 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:43:03.220929 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-111da73d-501a-4cb6-a569-3f34aea1dc01 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:03.221429 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-111da73d-501a-4cb6-a569-3f34aea1dc01 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:03.221874 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-111da73d-501a-4cb6-a569-3f34aea1dc01 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:03.244442 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-111da73d-501a-4cb6-a569-3f34aea1dc01 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:43:03.245773 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-111da73d-501a-4cb6-a569-3f34aea1dc01 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:43:03.315448 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:43:03.322275 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 538/1065] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:43:03 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 28 00:43:03.331357 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 528/1066] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Aug 28 00:43:02 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 505 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 28 00:43:03.337435 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-111da73d-501a-4cb6-a569-3f34aea1dc01 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:03.359874 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:43:03.483164 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-111da73d-501a-4cb6-a569-3f34aea1dc01 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:03.484483 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b5b77b7-5c1c-4ceb-85fa-b82e69adf24b None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:03.543318 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82b70cb4-2636-4b0b-add5-519600071eb5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:03.543957 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82b70cb4-2636-4b0b-add5-519600071eb5 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:03.544293 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82b70cb4-2636-4b0b-add5-519600071eb5 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:03.544738 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82b70cb4-2636-4b0b-add5-519600071eb5 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:03.555392 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82b70cb4-2636-4b0b-add5-519600071eb5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:43:03.556997 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82b70cb4-2636-4b0b-add5-519600071eb5 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:43:03.572643 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-82b70cb4-2636-4b0b-add5-519600071eb5 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:43:03.572643 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:43:03.572643 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:43:03.572643 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:43:03.572643 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:43:03.572643 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:43:03.572643 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:43:03.572643 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:43:03.572643 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:43:03.572643 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:43:03.572643 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:43:03.572643 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:43:03.572643 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 28 00:43:03.572643 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 28 00:43:03.572643 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 28 00:43:03.572643 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 28 00:43:03.573721 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:43:03.573721 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:43:03.573721 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:43:03.573721 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:43:03.573721 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:43:03.573721 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:43:03.573721 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:43:03.573721 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:43:03.573721 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:43:03.573721 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:43:03.573721 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:43:03.573721 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:43:03.573721 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:43:03.573721 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:43:03.573721 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:43:03.573721 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:43:03.573721 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 28 00:43:03.580725 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 28 00:43:03.580725 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:43:03.580725 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:43:03.580725 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 539/1067] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:43:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 237 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 28 00:43:03.580725 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82b70cb4-2636-4b0b-add5-519600071eb5 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:03.621840 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b5b77b7-5c1c-4ceb-85fa-b82e69adf24b None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:03.656346 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3defa494-ff50-44f9-9c38-cb60c3462601 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:03.657027 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3defa494-ff50-44f9-9c38-cb60c3462601 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:03.657164 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:43:03.657360 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3defa494-ff50-44f9-9c38-cb60c3462601 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:03.657573 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3defa494-ff50-44f9-9c38-cb60c3462601 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:03.668309 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3defa494-ff50-44f9-9c38-cb60c3462601 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 28 00:43:03.673323 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-3defa494-ff50-44f9-9c38-cb60c3462601 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 28 00:43:03.673323 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:43:03.673323 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:43:03.673323 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:43:03.673323 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:43:03.673323 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:43:03.673323 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:43:03.673323 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:43:03.673323 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:43:03.673323 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:43:03.673323 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:43:03.673323 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:43:03.673323 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 28 00:43:03.673323 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 28 00:43:03.673323 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 28 00:43:03.673323 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 28 00:43:03.675078 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 28 00:43:03.675078 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 28 00:43:03.675078 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 28 00:43:03.675078 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 28 00:43:03.675078 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:43:03.675078 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:43:03.675078 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:43:03.675078 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:43:03.675078 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:43:03.675078 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:43:03.675078 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:43:03.675078 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:43:03.675078 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:43:03.675078 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:43:03.675078 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:43:03.675078 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:43:03.676693 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:43:03.676693 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:43:03.676693 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:43:03.676693 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:43:03.676693 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 28 00:43:03.676693 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 28 00:43:03.676693 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 28 00:43:03.676693 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 28 00:43:03.676693 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 28 00:43:03.676693 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 28 00:43:03.676693 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 28 00:43:03.676693 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:43:03.676693 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 529/1068] 10.222.0.42 () {56 vars in 1120 bytes} [Sun Aug 28 00:43:03 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:43:03.676693 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3defa494-ff50-44f9-9c38-cb60c3462601 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:03.695126 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82b70cb4-2636-4b0b-add5-519600071eb5 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:03.730520 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:43:03.732468 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31d12fbe-fabf-4f75-9f65-e535e52997fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:03.733245 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d12fbe-fabf-4f75-9f65-e535e52997fb None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:03.733705 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d12fbe-fabf-4f75-9f65-e535e52997fb None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:03.734194 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d12fbe-fabf-4f75-9f65-e535e52997fb None admin] PATH_INFO: `/v3/domains` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:03.735017 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31d12fbe-fabf-4f75-9f65-e535e52997fb None admin] RBAC: Adding query filter params (name=Default) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:43:03.735498 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31d12fbe-fabf-4f75-9f65-e535e52997fb None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:43:03.737398 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31d12fbe-fabf-4f75-9f65-e535e52997fb None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:43:03.751645 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 540/1069] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Aug 28 00:43:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:43:03.752778 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31d12fbe-fabf-4f75-9f65-e535e52997fb None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:03.794024 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3defa494-ff50-44f9-9c38-cb60c3462601 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:03.839376 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-914b32b4-5f69-4e41-a509-d58ffeed5bfc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:03.840215 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-914b32b4-5f69-4e41-a509-d58ffeed5bfc None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:03.840630 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-914b32b4-5f69-4e41-a509-d58ffeed5bfc None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:03.841167 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-914b32b4-5f69-4e41-a509-d58ffeed5bfc None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:03.855504 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-914b32b4-5f69-4e41-a509-d58ffeed5bfc None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 28 00:43:03.862060 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-914b32b4-5f69-4e41-a509-d58ffeed5bfc None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 28 00:43:03.862060 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:43:03.862060 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:43:03.862060 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:43:03.862060 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:43:03.862060 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:43:03.862060 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:43:03.862060 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:43:03.862060 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:43:03.862060 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:43:03.862060 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:43:03.862060 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:43:03.862060 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 28 00:43:03.862060 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 28 00:43:03.862060 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 28 00:43:03.862060 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 28 00:43:03.863951 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 28 00:43:03.863951 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 28 00:43:03.863951 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 28 00:43:03.863951 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 28 00:43:03.863951 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:43:03.863951 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:43:03.863951 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:43:03.863951 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:43:03.863951 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:43:03.863951 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:43:03.863951 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:43:03.863951 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:43:03.863951 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:43:03.863951 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:43:03.863951 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:43:03.863951 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:43:03.864834 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:43:03.864834 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:43:03.864834 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:43:03.864834 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:43:03.864834 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 28 00:43:03.864834 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 28 00:43:03.864834 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 28 00:43:03.864834 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 28 00:43:03.864834 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 28 00:43:03.864834 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 28 00:43:03.864834 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 28 00:43:03.864834 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:43:03.864834 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 530/1070] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Aug 28 00:43:03 2022] GET /identity/v3/roles/admin => generated 83 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:43:03.870779 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-914b32b4-5f69-4e41-a509-d58ffeed5bfc None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:03.893356 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31d12fbe-fabf-4f75-9f65-e535e52997fb None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:03.934892 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:43:03.949062 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fb76dc5-f23f-45bd-961b-a24bd10191c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:03.949640 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb76dc5-f23f-45bd-961b-a24bd10191c2 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:03.949908 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb76dc5-f23f-45bd-961b-a24bd10191c2 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:03.950173 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb76dc5-f23f-45bd-961b-a24bd10191c2 None admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:03.951258 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fb76dc5-f23f-45bd-961b-a24bd10191c2 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:43:03.951258 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fb76dc5-f23f-45bd-961b-a24bd10191c2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:43:03.952552 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fb76dc5-f23f-45bd-961b-a24bd10191c2 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:43:03.962271 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 541/1071] 10.222.0.42 () {56 vars in 1115 bytes} [Sun Aug 28 00:43:03 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:43:03.968866 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fb76dc5-f23f-45bd-961b-a24bd10191c2 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:04.025938 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-914b32b4-5f69-4e41-a509-d58ffeed5bfc None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:04.042609 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:43:04.077875 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d3a859f-0def-4803-8565-36e2dea3c1f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:04.078507 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d3a859f-0def-4803-8565-36e2dea3c1f5 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:04.078786 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d3a859f-0def-4803-8565-36e2dea3c1f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:04.079059 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d3a859f-0def-4803-8565-36e2dea3c1f5 None admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:04.080052 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d3a859f-0def-4803-8565-36e2dea3c1f5 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:43:04.080355 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d3a859f-0def-4803-8565-36e2dea3c1f5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:43:04.081825 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d3a859f-0def-4803-8565-36e2dea3c1f5 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:43:04.091194 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 531/1072] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Aug 28 00:43:03 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:43:04.091471 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fb76dc5-f23f-45bd-961b-a24bd10191c2 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:04.098033 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d3a859f-0def-4803-8565-36e2dea3c1f5 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:04.132902 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc3a3fae-2d50-4398-a245-9e5f4e0d7e01 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:04.133583 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc3a3fae-2d50-4398-a245-9e5f4e0d7e01 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:04.133866 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc3a3fae-2d50-4398-a245-9e5f4e0d7e01 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:04.134131 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc3a3fae-2d50-4398-a245-9e5f4e0d7e01 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:04.146728 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc3a3fae-2d50-4398-a245-9e5f4e0d7e01 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:43:04.148188 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc3a3fae-2d50-4398-a245-9e5f4e0d7e01 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:43:04.163682 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-cc3a3fae-2d50-4398-a245-9e5f4e0d7e01 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:43:04.163682 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:43:04.163682 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:43:04.163682 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:43:04.163682 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:43:04.163682 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:43:04.163682 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:43:04.163682 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:43:04.163682 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:43:04.163682 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:43:04.163682 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:43:04.163682 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:43:04.163682 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 28 00:43:04.163682 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 28 00:43:04.163682 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 28 00:43:04.163682 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 28 00:43:04.165701 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:43:04.165701 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:43:04.165701 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:43:04.165701 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:43:04.165701 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:43:04.165701 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:43:04.165701 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:43:04.165701 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:43:04.165701 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:43:04.165701 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:43:04.165701 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:43:04.165701 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:43:04.165701 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:43:04.165701 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:43:04.165701 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:43:04.165701 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:43:04.165701 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 28 00:43:04.167433 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 28 00:43:04.167433 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:43:04.167433 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:43:04.167433 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 542/1073] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:43:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 28 00:43:04.171021 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc3a3fae-2d50-4398-a245-9e5f4e0d7e01 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:04.218614 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:43:04.221149 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d3a859f-0def-4803-8565-36e2dea3c1f5 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:04.258619 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1787162b-313b-4c1d-b290-16af86dab636 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:04.259444 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1787162b-313b-4c1d-b290-16af86dab636 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:04.259784 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1787162b-313b-4c1d-b290-16af86dab636 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:04.260121 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1787162b-313b-4c1d-b290-16af86dab636 None admin] PATH_INFO: `/v3/users/d95232208be944a6ae3ffbaf7007693d` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:04.261670 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1787162b-313b-4c1d-b290-16af86dab636 None admin] RBAC: Authorizing `identity:get_user(user_id=d95232208be944a6ae3ffbaf7007693d)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:43:04.263566 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1787162b-313b-4c1d-b290-16af86dab636 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:43:04.270858 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 532/1074] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Aug 28 00:43:04 2022] GET /identity/v3/users/d95232208be944a6ae3ffbaf7007693d => generated 285 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 28 00:43:04.272110 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1787162b-313b-4c1d-b290-16af86dab636 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:04.281148 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc3a3fae-2d50-4398-a245-9e5f4e0d7e01 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:04.318989 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:43:04.320594 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1b9b66a-45bd-425e-97ca-4407fc79f5d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:04.321663 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b9b66a-45bd-425e-97ca-4407fc79f5d1 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:04.322022 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b9b66a-45bd-425e-97ca-4407fc79f5d1 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:04.322446 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b9b66a-45bd-425e-97ca-4407fc79f5d1 None admin] PATH_INFO: `/v3/domains` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:04.323535 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1b9b66a-45bd-425e-97ca-4407fc79f5d1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:43:04.324030 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1b9b66a-45bd-425e-97ca-4407fc79f5d1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:43:04.326419 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1b9b66a-45bd-425e-97ca-4407fc79f5d1 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:43:04.346640 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 543/1075] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Aug 28 00:43:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 28 00:43:04.352399 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1b9b66a-45bd-425e-97ca-4407fc79f5d1 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:04.399128 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1787162b-313b-4c1d-b290-16af86dab636 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:04.436800 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebf84a32-0517-487f-8ff7-4358835726ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:04.437525 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebf84a32-0517-487f-8ff7-4358835726ae None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:04.437804 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebf84a32-0517-487f-8ff7-4358835726ae None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:04.438075 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebf84a32-0517-487f-8ff7-4358835726ae None admin] PATH_INFO: `/v3/projects/a661c03290754db3864357aaaffd5aa3` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:04.439239 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebf84a32-0517-487f-8ff7-4358835726ae None admin] RBAC: Authorizing `identity:get_project(project_id=a661c03290754db3864357aaaffd5aa3)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:43:04.440725 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebf84a32-0517-487f-8ff7-4358835726ae None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:43:04.442446 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 533/1076] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Aug 28 00:43:04 2022] GET /identity/v3/projects/a661c03290754db3864357aaaffd5aa3 => generated 311 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:43:04.448864 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebf84a32-0517-487f-8ff7-4358835726ae None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:04.457813 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1b9b66a-45bd-425e-97ca-4407fc79f5d1 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:04.459595 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:43:04.498321 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbaab6ef-83d7-4efd-88cc-8389de638ea8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:04.499030 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbaab6ef-83d7-4efd-88cc-8389de638ea8 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:04.499241 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbaab6ef-83d7-4efd-88cc-8389de638ea8 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:04.499510 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbaab6ef-83d7-4efd-88cc-8389de638ea8 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:04.515168 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbaab6ef-83d7-4efd-88cc-8389de638ea8 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:43:04.516648 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbaab6ef-83d7-4efd-88cc-8389de638ea8 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:43:04.530622 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-cbaab6ef-83d7-4efd-88cc-8389de638ea8 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 28 00:43:04.530622 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:43:04.530622 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:43:04.530622 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:43:04.530622 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:43:04.530622 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:43:04.530622 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:43:04.530622 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:43:04.530622 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:43:04.530622 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:43:04.530622 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:43:04.530622 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:43:04.530622 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 28 00:43:04.530622 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 28 00:43:04.530622 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 28 00:43:04.530622 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 28 00:43:04.531689 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:43:04.531689 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:43:04.531689 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 28 00:43:04.531689 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 28 00:43:04.531689 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 28 00:43:04.531689 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 28 00:43:04.531689 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:43:04.531689 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:43:04.531689 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:43:04.531689 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:43:04.531689 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:43:04.531689 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:43:04.531689 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:43:04.531689 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:43:04.531689 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:43:04.531689 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:43:04.531689 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:43:04.531689 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:43:04.532932 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:43:04.532932 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:43:04.532932 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 28 00:43:04.532932 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 28 00:43:04.532932 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 28 00:43:04.532932 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 28 00:43:04.532932 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 28 00:43:04.532932 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 28 00:43:04.532932 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 28 00:43:04.532932 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:43:04.532932 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 544/1077] 10.222.0.42 () {56 vars in 1093 bytes} [Sun Aug 28 00:43:04 2022] GET /identity/v3/users/nova => generated 82 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 28 00:43:04.538484 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbaab6ef-83d7-4efd-88cc-8389de638ea8 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:04.564955 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebf84a32-0517-487f-8ff7-4358835726ae None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:04.576501 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:43:04.605044 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae77261e-08c6-4d72-8d07-e8cb3f05bac2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:04.605758 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae77261e-08c6-4d72-8d07-e8cb3f05bac2 None admin] REQUEST_METHOD: `PUT` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:04.606021 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae77261e-08c6-4d72-8d07-e8cb3f05bac2 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:04.606278 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae77261e-08c6-4d72-8d07-e8cb3f05bac2 None admin] PATH_INFO: `/v3/projects/a661c03290754db3864357aaaffd5aa3/users/d95232208be944a6ae3ffbaf7007693d/roles/f08d6043261747e3b4763c744eef7dfb` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:04.608326 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae77261e-08c6-4d72-8d07-e8cb3f05bac2 None admin] RBAC: Authorizing `identity:create_grant(project_id=a661c03290754db3864357aaaffd5aa3, user_id=d95232208be944a6ae3ffbaf7007693d, role_id=f08d6043261747e3b4763c744eef7dfb)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:43:04.609833 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae77261e-08c6-4d72-8d07-e8cb3f05bac2 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:43:04.625047 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 534/1078] 10.222.0.42 () {58 vars in 1439 bytes} [Sun Aug 28 00:43:04 2022] PUT /identity/v3/projects/a661c03290754db3864357aaaffd5aa3/users/d95232208be944a6ae3ffbaf7007693d/roles/f08d6043261747e3b4763c744eef7dfb => generated 0 bytes in 177 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 28 00:43:04.654457 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbaab6ef-83d7-4efd-88cc-8389de638ea8 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:04.694841 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f68b64fb-e0d6-413d-8fc6-427ad1e0712a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:04.695462 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f68b64fb-e0d6-413d-8fc6-427ad1e0712a None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:04.695742 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f68b64fb-e0d6-413d-8fc6-427ad1e0712a None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:04.696016 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f68b64fb-e0d6-413d-8fc6-427ad1e0712a None admin] PATH_INFO: `/v3/users` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:04.696781 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f68b64fb-e0d6-413d-8fc6-427ad1e0712a None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:43:04.697126 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f68b64fb-e0d6-413d-8fc6-427ad1e0712a None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:43:04.698644 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f68b64fb-e0d6-413d-8fc6-427ad1e0712a None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:43:04.720561 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 545/1079] 10.222.0.42 () {56 vars in 1133 bytes} [Sun Aug 28 00:43:04 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 28 00:43:04.727099 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae77261e-08c6-4d72-8d07-e8cb3f05bac2 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:04.870694 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae77261e-08c6-4d72-8d07-e8cb3f05bac2 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:04.912821 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-859e7a9b-2ab2-49e0-a98f-6b9d86e92f16 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:04.913433 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-859e7a9b-2ab2-49e0-a98f-6b9d86e92f16 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:04.913713 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-859e7a9b-2ab2-49e0-a98f-6b9d86e92f16 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:04.913986 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-859e7a9b-2ab2-49e0-a98f-6b9d86e92f16 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:04.926165 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-859e7a9b-2ab2-49e0-a98f-6b9d86e92f16 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:43:04.927681 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-859e7a9b-2ab2-49e0-a98f-6b9d86e92f16 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:43:04.943524 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-859e7a9b-2ab2-49e0-a98f-6b9d86e92f16 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:43:04.943524 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:43:04.943524 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:43:04.943524 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:43:04.943524 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:43:04.943524 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:43:04.943524 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:43:04.943524 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:43:04.943524 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:43:04.943524 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:43:04.943524 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:43:04.943524 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:43:04.943524 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 28 00:43:04.943524 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 28 00:43:04.943524 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 28 00:43:04.943524 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 28 00:43:04.945550 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:43:04.945550 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:43:04.945550 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:43:04.945550 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:43:04.945550 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:43:04.945550 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:43:04.945550 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:43:04.945550 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:43:04.945550 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:43:04.945550 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:43:04.945550 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:43:04.945550 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:43:04.945550 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:43:04.945550 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:43:04.945550 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:43:04.945550 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:43:04.945550 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 28 00:43:04.947431 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 28 00:43:04.947431 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:43:04.947431 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:43:04.949188 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 535/1080] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:43:04 2022] GET /identity/v3/domains/Default => generated 87 bytes in 223 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:43:04.950968 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f68b64fb-e0d6-413d-8fc6-427ad1e0712a None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:04.975967 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:43:05.060499 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f68b64fb-e0d6-413d-8fc6-427ad1e0712a None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:05.097039 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bca34462-1fd9-495c-8e11-e1e9563fb76e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:05.097646 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca34462-1fd9-495c-8e11-e1e9563fb76e None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:05.097925 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca34462-1fd9-495c-8e11-e1e9563fb76e None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:05.098196 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca34462-1fd9-495c-8e11-e1e9563fb76e None admin] PATH_INFO: `/v3/domains` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:05.098861 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bca34462-1fd9-495c-8e11-e1e9563fb76e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:43:05.099164 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bca34462-1fd9-495c-8e11-e1e9563fb76e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:43:05.100556 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bca34462-1fd9-495c-8e11-e1e9563fb76e None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:43:05.124483 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 546/1081] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Aug 28 00:43:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:43:05.125808 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-859e7a9b-2ab2-49e0-a98f-6b9d86e92f16 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:05.149622 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:43:05.257431 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-859e7a9b-2ab2-49e0-a98f-6b9d86e92f16 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:05.297095 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a6ce21c-a890-4412-a194-a0bfd7d857b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:05.297730 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a6ce21c-a890-4412-a194-a0bfd7d857b2 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:05.298009 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a6ce21c-a890-4412-a194-a0bfd7d857b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:05.298278 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a6ce21c-a890-4412-a194-a0bfd7d857b2 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:05.305697 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a6ce21c-a890-4412-a194-a0bfd7d857b2 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:43:05.307160 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a6ce21c-a890-4412-a194-a0bfd7d857b2 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:43:05.319396 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-0a6ce21c-a890-4412-a194-a0bfd7d857b2 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 28 00:43:05.319396 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:43:05.319396 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:43:05.319396 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:43:05.319396 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:43:05.319396 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:43:05.319396 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:43:05.319396 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:43:05.319396 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:43:05.319396 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:43:05.319396 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:43:05.319396 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:43:05.319396 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 28 00:43:05.319396 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 28 00:43:05.319396 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 28 00:43:05.321026 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 28 00:43:05.321026 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:43:05.321026 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:43:05.321026 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:43:05.321026 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:43:05.321026 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:43:05.321026 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:43:05.321026 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:43:05.321026 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:43:05.321026 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:43:05.321026 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:43:05.321026 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:43:05.321026 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:43:05.321026 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:43:05.321026 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:43:05.321026 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:43:05.321026 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:43:05.322679 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 28 00:43:05.322679 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 28 00:43:05.322679 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 28 00:43:05.322679 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 28 00:43:05.322679 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 28 00:43:05.322679 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 28 00:43:05.322679 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 28 00:43:05.322679 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:43:05.325250 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 536/1082] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Aug 28 00:43:05 2022] GET /identity/v3/projects/service => generated 88 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 28 00:43:05.326789 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bca34462-1fd9-495c-8e11-e1e9563fb76e None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:05.414694 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:43:05.443156 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bca34462-1fd9-495c-8e11-e1e9563fb76e None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:05.479123 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-833e3ce3-65bc-4047-924b-1d9d3ddad301 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:05.479738 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-833e3ce3-65bc-4047-924b-1d9d3ddad301 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:05.480018 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-833e3ce3-65bc-4047-924b-1d9d3ddad301 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:05.480283 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-833e3ce3-65bc-4047-924b-1d9d3ddad301 None admin] PATH_INFO: `/v3/projects` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:05.480998 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-833e3ce3-65bc-4047-924b-1d9d3ddad301 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:43:05.481305 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-833e3ce3-65bc-4047-924b-1d9d3ddad301 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:43:05.482786 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-833e3ce3-65bc-4047-924b-1d9d3ddad301 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:43:05.497885 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 547/1083] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Aug 28 00:43:05 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:43:05.504161 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a6ce21c-a890-4412-a194-a0bfd7d857b2 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:05.593380 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:43:05.622693 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a6ce21c-a890-4412-a194-a0bfd7d857b2 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:05.679152 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78f6bef8-6c23-4308-8505-3048bf5f15f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:05.679980 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78f6bef8-6c23-4308-8505-3048bf5f15f3 None admin] REQUEST_METHOD: `PUT` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:05.680327 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78f6bef8-6c23-4308-8505-3048bf5f15f3 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:05.680688 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78f6bef8-6c23-4308-8505-3048bf5f15f3 None admin] PATH_INFO: `/v3/projects/f53e6fdb369b480fbebb4086a505df67/users/168b07df50c24593ba73dbde5b037aed/roles/2c6107d249594bd3abf791034b71e98f` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:05.683649 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78f6bef8-6c23-4308-8505-3048bf5f15f3 None admin] RBAC: Authorizing `identity:create_grant(project_id=f53e6fdb369b480fbebb4086a505df67, user_id=168b07df50c24593ba73dbde5b037aed, role_id=2c6107d249594bd3abf791034b71e98f)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:43:05.685343 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78f6bef8-6c23-4308-8505-3048bf5f15f3 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:43:05.703730 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 537/1084] 10.222.0.42 () {58 vars in 1439 bytes} [Sun Aug 28 00:43:05 2022] PUT /identity/v3/projects/f53e6fdb369b480fbebb4086a505df67/users/168b07df50c24593ba73dbde5b037aed/roles/2c6107d249594bd3abf791034b71e98f => generated 0 bytes in 200 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:43:06.586707 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-773d1115-b123-4f11-a293-8e144e95da9c None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:06.587314 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-773d1115-b123-4f11-a293-8e144e95da9c None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:06.587314 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-773d1115-b123-4f11-a293-8e144e95da9c None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:06.588612 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 548/1085] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:43:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:43:06.594577 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c514fdfd-3e5d-4bba-bc9b-a0fa9286bdf5 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:06.595105 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c514fdfd-3e5d-4bba-bc9b-a0fa9286bdf5 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:06.595105 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c514fdfd-3e5d-4bba-bc9b-a0fa9286bdf5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:06.615371 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-c514fdfd-3e5d-4bba-bc9b-a0fa9286bdf5 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:43:06.616742 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-c514fdfd-3e5d-4bba-bc9b-a0fa9286bdf5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:43:06.689978 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 538/1086] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:43:06 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 28 00:43:06.962150 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-773d1115-b123-4f11-a293-8e144e95da9c None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:07.069631 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-773d1115-b123-4f11-a293-8e144e95da9c None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:07.106935 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4542efa1-99c5-49d4-8c0a-f4394ac01525 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:07.107577 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4542efa1-99c5-49d4-8c0a-f4394ac01525 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:07.107856 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4542efa1-99c5-49d4-8c0a-f4394ac01525 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:07.108125 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4542efa1-99c5-49d4-8c0a-f4394ac01525 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:07.121149 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:43:07.125700 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4542efa1-99c5-49d4-8c0a-f4394ac01525 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 28 00:43:07.131019 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-4542efa1-99c5-49d4-8c0a-f4394ac01525 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 28 00:43:07.131019 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:43:07.131019 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:43:07.131019 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:43:07.131019 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:43:07.131019 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:43:07.131019 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:43:07.131019 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:43:07.131019 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:43:07.131019 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:43:07.131019 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:43:07.131019 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:43:07.131019 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 28 00:43:07.131019 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 28 00:43:07.131019 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 28 00:43:07.131019 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 28 00:43:07.133001 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 28 00:43:07.133001 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 28 00:43:07.133001 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 28 00:43:07.133001 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 28 00:43:07.133001 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:43:07.133001 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:43:07.133001 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:43:07.133001 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:43:07.133001 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:43:07.133001 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:43:07.133001 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:43:07.133001 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:43:07.133001 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:43:07.133001 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:43:07.133001 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:43:07.133001 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:43:07.133960 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:43:07.133960 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:43:07.133960 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:43:07.133960 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:43:07.133960 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 28 00:43:07.133960 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 28 00:43:07.133960 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 28 00:43:07.133960 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 28 00:43:07.133960 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 28 00:43:07.133960 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 28 00:43:07.133960 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 28 00:43:07.133960 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:43:07.133960 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 549/1087] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Aug 28 00:43:06 2022] GET /identity/v3/roles/admin => generated 83 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:43:07.139272 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c514fdfd-3e5d-4bba-bc9b-a0fa9286bdf5 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:07.213316 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:43:07.247768 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c514fdfd-3e5d-4bba-bc9b-a0fa9286bdf5 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:07.297342 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e27f39c-7f85-4ab1-8fda-c87ad1d6ed9d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:07.298358 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e27f39c-7f85-4ab1-8fda-c87ad1d6ed9d None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:07.298838 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e27f39c-7f85-4ab1-8fda-c87ad1d6ed9d None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:07.299312 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e27f39c-7f85-4ab1-8fda-c87ad1d6ed9d None admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:07.300506 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e27f39c-7f85-4ab1-8fda-c87ad1d6ed9d None admin] RBAC: Adding query filter params (name=admin) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:43:07.301057 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e27f39c-7f85-4ab1-8fda-c87ad1d6ed9d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:43:07.303608 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e27f39c-7f85-4ab1-8fda-c87ad1d6ed9d None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:43:07.318507 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 539/1088] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Aug 28 00:43:07 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:43:07.324476 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4542efa1-99c5-49d4-8c0a-f4394ac01525 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:07.430922 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4542efa1-99c5-49d4-8c0a-f4394ac01525 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:07.467149 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40040642-4f19-4b57-abcb-35cf2d631cf0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:07.467815 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40040642-4f19-4b57-abcb-35cf2d631cf0 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:07.468187 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40040642-4f19-4b57-abcb-35cf2d631cf0 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:07.468546 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40040642-4f19-4b57-abcb-35cf2d631cf0 None admin] PATH_INFO: `/v3/users/d95232208be944a6ae3ffbaf7007693d` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:07.469854 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40040642-4f19-4b57-abcb-35cf2d631cf0 None admin] RBAC: Authorizing `identity:get_user(user_id=d95232208be944a6ae3ffbaf7007693d)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:43:07.471404 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40040642-4f19-4b57-abcb-35cf2d631cf0 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:43:07.473000 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 550/1089] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Aug 28 00:43:07 2022] GET /identity/v3/users/d95232208be944a6ae3ffbaf7007693d => generated 285 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:43:07.479214 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e27f39c-7f85-4ab1-8fda-c87ad1d6ed9d None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:07.587220 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e27f39c-7f85-4ab1-8fda-c87ad1d6ed9d None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:07.608142 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd19f39-34a1-4f44-ba5a-1cb0abaea4ab None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:07.608876 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd19f39-34a1-4f44-ba5a-1cb0abaea4ab None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:07.608876 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd19f39-34a1-4f44-ba5a-1cb0abaea4ab None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:07.610234 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 551/1090] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:43:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 28 00:43:07.615714 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea6d5cfa-5cdc-4c6a-8be2-234256057b65 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:07.616001 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea6d5cfa-5cdc-4c6a-8be2-234256057b65 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:07.616268 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea6d5cfa-5cdc-4c6a-8be2-234256057b65 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:07.618564 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:43:07.624348 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0f7cfbe-2ebf-4481-a00c-1a2831a2e5b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:07.625127 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0f7cfbe-2ebf-4481-a00c-1a2831a2e5b6 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:07.625407 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0f7cfbe-2ebf-4481-a00c-1a2831a2e5b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:07.625679 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0f7cfbe-2ebf-4481-a00c-1a2831a2e5b6 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:07.638351 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-ea6d5cfa-5cdc-4c6a-8be2-234256057b65 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:43:07.640553 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-ea6d5cfa-5cdc-4c6a-8be2-234256057b65 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:43:07.703484 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0f7cfbe-2ebf-4481-a00c-1a2831a2e5b6 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:43:07.705762 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0f7cfbe-2ebf-4481-a00c-1a2831a2e5b6 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:43:07.718894 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:43:07.726562 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 552/1091] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:43:07 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 28 00:43:07.828706 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 540/1092] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Aug 28 00:43:07 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 350 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 28 00:43:07.835178 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea6d5cfa-5cdc-4c6a-8be2-234256057b65 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:07.946348 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea6d5cfa-5cdc-4c6a-8be2-234256057b65 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:07.986080 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a45e002f-e7d3-4a38-8924-bb262f1070fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:07.986634 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a45e002f-e7d3-4a38-8924-bb262f1070fb None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:07.986917 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a45e002f-e7d3-4a38-8924-bb262f1070fb None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:07.987193 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a45e002f-e7d3-4a38-8924-bb262f1070fb None admin] PATH_INFO: `/v3/projects/a661c03290754db3864357aaaffd5aa3` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:07.988387 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a45e002f-e7d3-4a38-8924-bb262f1070fb None admin] RBAC: Authorizing `identity:get_project(project_id=a661c03290754db3864357aaaffd5aa3)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:43:07.989911 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a45e002f-e7d3-4a38-8924-bb262f1070fb None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:43:07.991645 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 553/1093] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Aug 28 00:43:07 2022] GET /identity/v3/projects/a661c03290754db3864357aaaffd5aa3 => generated 311 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:43:07.992649 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0f7cfbe-2ebf-4481-a00c-1a2831a2e5b6 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:07.998573 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a45e002f-e7d3-4a38-8924-bb262f1070fb None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:08.027894 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:43:08.076903 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:43:08.109603 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0f7cfbe-2ebf-4481-a00c-1a2831a2e5b6 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:08.126035 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a45e002f-e7d3-4a38-8924-bb262f1070fb None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:08.152931 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd6a8447-7653-4a46-acdc-7859d3aca244 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:08.153900 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd6a8447-7653-4a46-acdc-7859d3aca244 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:08.154245 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd6a8447-7653-4a46-acdc-7859d3aca244 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:08.154570 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd6a8447-7653-4a46-acdc-7859d3aca244 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:08.163321 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2bab585-24f8-4007-a1d3-df453dcaa066 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:08.164237 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2bab585-24f8-4007-a1d3-df453dcaa066 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:08.164510 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2bab585-24f8-4007-a1d3-df453dcaa066 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:08.164778 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2bab585-24f8-4007-a1d3-df453dcaa066 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:08.165537 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2bab585-24f8-4007-a1d3-df453dcaa066 None admin] RBAC: Adding query filter params (role.id=f08d6043261747e3b4763c744eef7dfb, scope.project.id=a661c03290754db3864357aaaffd5aa3, user.id=d95232208be944a6ae3ffbaf7007693d) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:43:08.165992 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2bab585-24f8-4007-a1d3-df453dcaa066 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:43:08.166730 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd6a8447-7653-4a46-acdc-7859d3aca244 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 28 00:43:08.167435 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2bab585-24f8-4007-a1d3-df453dcaa066 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:43:08.172194 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-dd6a8447-7653-4a46-acdc-7859d3aca244 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 28 00:43:08.172194 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:43:08.172194 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:43:08.172194 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:43:08.172194 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:43:08.172194 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:43:08.172194 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:43:08.172194 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:43:08.172194 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:43:08.172194 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:43:08.172194 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:43:08.172194 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:43:08.172194 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 28 00:43:08.172194 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 28 00:43:08.172194 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 28 00:43:08.172194 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 28 00:43:08.173317 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 28 00:43:08.173317 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 28 00:43:08.173317 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 28 00:43:08.173317 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 28 00:43:08.173317 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:43:08.173317 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:43:08.173317 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:43:08.173317 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:43:08.173317 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:43:08.173317 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:43:08.173317 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:43:08.173317 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:43:08.173317 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:43:08.173317 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:43:08.173317 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:43:08.173317 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:43:08.174182 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:43:08.174182 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:43:08.174182 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:43:08.174182 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:43:08.174182 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 28 00:43:08.174182 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 28 00:43:08.174182 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 28 00:43:08.174182 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 28 00:43:08.174182 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 28 00:43:08.174182 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 28 00:43:08.174182 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Aug 28 00:43:08.174182 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:43:08.174182 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 541/1095] 10.222.0.42 () {56 vars in 1120 bytes} [Sun Aug 28 00:43:07 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 28 00:43:08.174898 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 554/1095] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Aug 28 00:43:07 2022] GET /identity/v3/role_assignments?user.id=d95232208be944a6ae3ffbaf7007693d&scope.project.id=a661c03290754db3864357aaaffd5aa3&role.id=f08d6043261747e3b4763c744eef7dfb => generated 608 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 28 00:43:08.182763 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd6a8447-7653-4a46-acdc-7859d3aca244 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:08.344851 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:43:08.361187 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd6a8447-7653-4a46-acdc-7859d3aca244 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:08.414048 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a362d2ac-8398-4322-b215-aafc936f787c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:08.414987 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a362d2ac-8398-4322-b215-aafc936f787c None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:08.415392 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a362d2ac-8398-4322-b215-aafc936f787c None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:08.415731 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a362d2ac-8398-4322-b215-aafc936f787c None admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:08.416460 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a362d2ac-8398-4322-b215-aafc936f787c None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:43:08.416885 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a362d2ac-8398-4322-b215-aafc936f787c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:43:08.418771 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a362d2ac-8398-4322-b215-aafc936f787c None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:43:08.430119 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 542/1096] 10.222.0.42 () {56 vars in 1115 bytes} [Sun Aug 28 00:43:08 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 248 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:43:08.437203 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2bab585-24f8-4007-a1d3-df453dcaa066 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:08.553737 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2bab585-24f8-4007-a1d3-df453dcaa066 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:08.585572 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:43:08.593840 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c1c2c4f-2df6-4385-8e57-886daf3d2385 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:08.594486 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c1c2c4f-2df6-4385-8e57-886daf3d2385 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:08.594766 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c1c2c4f-2df6-4385-8e57-886daf3d2385 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:08.595086 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c1c2c4f-2df6-4385-8e57-886daf3d2385 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:08.610654 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c1c2c4f-2df6-4385-8e57-886daf3d2385 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:43:08.612171 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c1c2c4f-2df6-4385-8e57-886daf3d2385 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:43:08.630407 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-8c1c2c4f-2df6-4385-8e57-886daf3d2385 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:43:08.630407 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:43:08.630407 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:43:08.630407 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:43:08.630407 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:43:08.630407 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:43:08.630407 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:43:08.630407 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:43:08.630407 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:43:08.630407 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:43:08.630407 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:43:08.630407 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:43:08.630407 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 28 00:43:08.630407 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 28 00:43:08.630407 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 28 00:43:08.630407 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 28 00:43:08.631621 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:43:08.631621 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:43:08.631621 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:43:08.631621 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:43:08.631621 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:43:08.631621 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:43:08.631621 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:43:08.631621 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:43:08.631621 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:43:08.631621 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:43:08.631621 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:43:08.631621 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:43:08.631621 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:43:08.631621 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:43:08.631621 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:43:08.631621 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:43:08.631621 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 28 00:43:08.634158 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 28 00:43:08.634158 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:43:08.634158 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:43:08.634158 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 555/1097] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:43:08 2022] GET /identity/v3/domains/Default => generated 87 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:43:08.639006 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a362d2ac-8398-4322-b215-aafc936f787c None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:08.748667 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a362d2ac-8398-4322-b215-aafc936f787c None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:08.786171 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a538f2d1-75ff-4a53-a2e0-634fe3480755 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:08.786808 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a538f2d1-75ff-4a53-a2e0-634fe3480755 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:08.787089 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a538f2d1-75ff-4a53-a2e0-634fe3480755 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:08.787364 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a538f2d1-75ff-4a53-a2e0-634fe3480755 None admin] PATH_INFO: `/v3/domains` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:08.788007 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a538f2d1-75ff-4a53-a2e0-634fe3480755 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:43:08.788307 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a538f2d1-75ff-4a53-a2e0-634fe3480755 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:43:08.789743 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a538f2d1-75ff-4a53-a2e0-634fe3480755 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:43:08.802954 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 543/1098] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Aug 28 00:43:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 28 00:43:08.808810 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c1c2c4f-2df6-4385-8e57-886daf3d2385 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:08.922900 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c1c2c4f-2df6-4385-8e57-886daf3d2385 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:08.959072 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8569f5e6-a894-46fa-8ffb-f406969bf309 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:08.959727 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8569f5e6-a894-46fa-8ffb-f406969bf309 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:08.960002 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8569f5e6-a894-46fa-8ffb-f406969bf309 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:08.960274 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8569f5e6-a894-46fa-8ffb-f406969bf309 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:08.968736 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8569f5e6-a894-46fa-8ffb-f406969bf309 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:43:08.970255 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8569f5e6-a894-46fa-8ffb-f406969bf309 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:43:08.982999 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-8569f5e6-a894-46fa-8ffb-f406969bf309 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Aug 28 00:43:08.982999 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:43:08.982999 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:43:08.982999 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:43:08.982999 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:43:08.982999 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:43:08.982999 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:43:08.982999 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:43:08.982999 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:43:08.982999 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:43:08.982999 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:43:08.982999 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:43:08.982999 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 28 00:43:08.982999 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 28 00:43:08.982999 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 28 00:43:08.982999 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 28 00:43:08.984739 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:43:08.984739 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:43:08.984739 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 28 00:43:08.984739 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 28 00:43:08.984739 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 28 00:43:08.984739 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 28 00:43:08.984739 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:43:08.984739 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:43:08.984739 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:43:08.984739 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:43:08.984739 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:43:08.984739 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:43:08.984739 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:43:08.984739 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:43:08.984739 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:43:08.984739 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:43:08.984739 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:43:08.984739 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:43:08.985759 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:43:08.985759 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:43:08.985759 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 28 00:43:08.985759 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 28 00:43:08.985759 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 28 00:43:08.985759 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 28 00:43:08.985759 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 28 00:43:08.985759 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 28 00:43:08.985759 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Aug 28 00:43:08.985759 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:43:08.985759 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 556/1099] 10.222.0.42 () {56 vars in 1093 bytes} [Sun Aug 28 00:43:08 2022] GET /identity/v3/users/nova => generated 82 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:43:08.990679 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a538f2d1-75ff-4a53-a2e0-634fe3480755 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:09.008893 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:43:09.104790 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a538f2d1-75ff-4a53-a2e0-634fe3480755 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:09.157836 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8045e87-913c-4bec-9ef2-6785f60ee624 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:09.158492 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8045e87-913c-4bec-9ef2-6785f60ee624 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:09.158767 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8045e87-913c-4bec-9ef2-6785f60ee624 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:09.159041 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8045e87-913c-4bec-9ef2-6785f60ee624 None admin] PATH_INFO: `/v3/users` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:09.159782 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8045e87-913c-4bec-9ef2-6785f60ee624 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:43:09.160085 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8045e87-913c-4bec-9ef2-6785f60ee624 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:43:09.161592 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8045e87-913c-4bec-9ef2-6785f60ee624 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:43:09.181243 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 544/1100] 10.222.0.42 () {56 vars in 1133 bytes} [Sun Aug 28 00:43:08 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:43:09.187204 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8569f5e6-a894-46fa-8ffb-f406969bf309 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:09.235122 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:43:09.304402 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8569f5e6-a894-46fa-8ffb-f406969bf309 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:09.362597 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42060b3f-05a0-4c55-9d37-c6cc1b0705b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:09.363545 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42060b3f-05a0-4c55-9d37-c6cc1b0705b7 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:09.363966 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42060b3f-05a0-4c55-9d37-c6cc1b0705b7 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:09.364417 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42060b3f-05a0-4c55-9d37-c6cc1b0705b7 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:09.479848 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42060b3f-05a0-4c55-9d37-c6cc1b0705b7 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:43:09.481228 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42060b3f-05a0-4c55-9d37-c6cc1b0705b7 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:43:09.526281 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:43:09.588987 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 557/1101] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Aug 28 00:43:09 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 402 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 28 00:43:09.595084 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8045e87-913c-4bec-9ef2-6785f60ee624 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:09.669472 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:43:09.705794 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8045e87-913c-4bec-9ef2-6785f60ee624 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:09.742742 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c0ab7f5-6098-403c-b270-fd528b740396 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:09.743299 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c0ab7f5-6098-403c-b270-fd528b740396 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:09.743578 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c0ab7f5-6098-403c-b270-fd528b740396 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:09.743839 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c0ab7f5-6098-403c-b270-fd528b740396 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:09.756071 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c0ab7f5-6098-403c-b270-fd528b740396 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:43:09.757685 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c0ab7f5-6098-403c-b270-fd528b740396 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:43:09.772595 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-7c0ab7f5-6098-403c-b270-fd528b740396 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:43:09.772595 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:43:09.772595 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:43:09.772595 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:43:09.772595 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:43:09.772595 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:43:09.772595 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:43:09.772595 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:43:09.772595 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:43:09.772595 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:43:09.772595 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:43:09.772595 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:43:09.772595 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 28 00:43:09.772595 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 28 00:43:09.772595 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 28 00:43:09.772595 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 28 00:43:09.774336 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:43:09.774336 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:43:09.774336 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:43:09.774336 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:43:09.774336 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:43:09.774336 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:43:09.774336 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:43:09.774336 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:43:09.774336 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:43:09.774336 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:43:09.774336 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:43:09.774336 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:43:09.774336 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:43:09.774336 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:43:09.774336 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:43:09.774336 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:43:09.774336 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 28 00:43:09.776074 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 28 00:43:09.776074 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:43:09.776074 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:43:09.776074 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 545/1102] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:43:09 2022] GET /identity/v3/domains/Default => generated 87 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:43:09.780760 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42060b3f-05a0-4c55-9d37-c6cc1b0705b7 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:09.884698 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42060b3f-05a0-4c55-9d37-c6cc1b0705b7 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:09.922966 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69cb4621-a3e8-48ba-81f3-a471b93ecf2c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:09.923587 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69cb4621-a3e8-48ba-81f3-a471b93ecf2c None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:09.923857 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69cb4621-a3e8-48ba-81f3-a471b93ecf2c None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:09.924138 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69cb4621-a3e8-48ba-81f3-a471b93ecf2c None admin] PATH_INFO: `/v3/domains` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:09.924791 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69cb4621-a3e8-48ba-81f3-a471b93ecf2c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:43:09.925118 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69cb4621-a3e8-48ba-81f3-a471b93ecf2c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:43:09.926505 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69cb4621-a3e8-48ba-81f3-a471b93ecf2c None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:43:09.938778 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 558/1103] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Aug 28 00:43:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:43:09.944611 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c0ab7f5-6098-403c-b270-fd528b740396 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:10.050748 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c0ab7f5-6098-403c-b270-fd528b740396 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:10.075506 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:43:10.087537 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae41fa8b-ce91-4adc-a655-acb67bdaa0ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:10.088169 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae41fa8b-ce91-4adc-a655-acb67bdaa0ed None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:10.088451 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae41fa8b-ce91-4adc-a655-acb67bdaa0ed None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:10.088723 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae41fa8b-ce91-4adc-a655-acb67bdaa0ed None admin] PATH_INFO: `/v3/projects/service` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:10.095723 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae41fa8b-ce91-4adc-a655-acb67bdaa0ed None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:43:10.097244 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae41fa8b-ce91-4adc-a655-acb67bdaa0ed None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:43:10.109163 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-ae41fa8b-ce91-4adc-a655-acb67bdaa0ed None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 28 00:43:10.109163 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:43:10.109163 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:43:10.109163 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:43:10.109163 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:43:10.109163 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:43:10.109163 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:43:10.109163 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:43:10.109163 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:43:10.109163 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:43:10.109163 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:43:10.109163 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:43:10.109163 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 28 00:43:10.109163 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 28 00:43:10.109163 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 28 00:43:10.117939 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 28 00:43:10.117939 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:43:10.117939 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:43:10.117939 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:43:10.117939 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:43:10.117939 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:43:10.117939 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:43:10.117939 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:43:10.117939 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:43:10.117939 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:43:10.117939 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:43:10.117939 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:43:10.117939 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:43:10.117939 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:43:10.117939 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:43:10.117939 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:43:10.117939 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:43:10.119658 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 28 00:43:10.119658 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 28 00:43:10.119658 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 28 00:43:10.119658 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 28 00:43:10.119658 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 28 00:43:10.119658 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 28 00:43:10.119658 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 28 00:43:10.119658 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:43:10.119658 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 546/1104] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Aug 28 00:43:09 2022] GET /identity/v3/projects/service => generated 88 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:43:10.122000 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69cb4621-a3e8-48ba-81f3-a471b93ecf2c None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:10.123683 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:43:10.219473 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-507230b4-6dc6-4c72-a1b6-7302c25d9d3f None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:10.219919 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-507230b4-6dc6-4c72-a1b6-7302c25d9d3f None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:10.220357 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-507230b4-6dc6-4c72-a1b6-7302c25d9d3f None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:10.222675 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 547/1105] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:43:10 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 28 00:43:10.228514 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94bc1409-17d7-4726-971e-149882ca1f9d None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:10.228788 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94bc1409-17d7-4726-971e-149882ca1f9d None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:10.229091 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94bc1409-17d7-4726-971e-149882ca1f9d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:10.236922 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69cb4621-a3e8-48ba-81f3-a471b93ecf2c None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:10.248465 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-94bc1409-17d7-4726-971e-149882ca1f9d None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:43:10.249750 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-94bc1409-17d7-4726-971e-149882ca1f9d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:43:10.274487 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fffdc521-b20d-4364-aec4-ce69258ae605 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:10.275298 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fffdc521-b20d-4364-aec4-ce69258ae605 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:10.275664 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fffdc521-b20d-4364-aec4-ce69258ae605 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:10.276112 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fffdc521-b20d-4364-aec4-ce69258ae605 None admin] PATH_INFO: `/v3/projects` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:10.276931 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fffdc521-b20d-4364-aec4-ce69258ae605 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:43:10.277411 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fffdc521-b20d-4364-aec4-ce69258ae605 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:43:10.279048 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fffdc521-b20d-4364-aec4-ce69258ae605 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:43:10.297367 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 559/1106] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Aug 28 00:43:10 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:43:10.299112 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fffdc521-b20d-4364-aec4-ce69258ae605 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:10.326252 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 548/1107] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:43:10 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 28 00:43:10.362567 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:43:10.402822 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fffdc521-b20d-4364-aec4-ce69258ae605 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:10.438083 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f3b7bc0-513d-4998-b311-b77b7d5a5c71 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:10.438691 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f3b7bc0-513d-4998-b311-b77b7d5a5c71 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:10.438970 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f3b7bc0-513d-4998-b311-b77b7d5a5c71 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:10.439237 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f3b7bc0-513d-4998-b311-b77b7d5a5c71 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:10.439959 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f3b7bc0-513d-4998-b311-b77b7d5a5c71 None admin] RBAC: Adding query filter params (role.id=2c6107d249594bd3abf791034b71e98f, scope.project.id=f53e6fdb369b480fbebb4086a505df67, user.id=168b07df50c24593ba73dbde5b037aed) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:43:10.440262 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f3b7bc0-513d-4998-b311-b77b7d5a5c71 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:43:10.441741 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f3b7bc0-513d-4998-b311-b77b7d5a5c71 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:43:10.447729 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 560/1108] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Aug 28 00:43:10 2022] GET /identity/v3/role_assignments?user.id=168b07df50c24593ba73dbde5b037aed&scope.project.id=f53e6fdb369b480fbebb4086a505df67&role.id=2c6107d249594bd3abf791034b71e98f => generated 608 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:43:10.828603 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94bc1409-17d7-4726-971e-149882ca1f9d None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:10.964362 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94bc1409-17d7-4726-971e-149882ca1f9d None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:10.980187 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:43:11.008644 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b1f9997-8bd4-4397-8cce-19ec20087e38 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:11.009757 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b1f9997-8bd4-4397-8cce-19ec20087e38 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:11.010252 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b1f9997-8bd4-4397-8cce-19ec20087e38 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:11.010723 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b1f9997-8bd4-4397-8cce-19ec20087e38 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:11.012654 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b1f9997-8bd4-4397-8cce-19ec20087e38 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:43:11.015403 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b1f9997-8bd4-4397-8cce-19ec20087e38 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:43:11.018361 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 549/1109] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:43:10 2022] GET /identity/v3/domains/default => generated 199 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:43:11.024291 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f3b7bc0-513d-4998-b311-b77b7d5a5c71 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:11.142778 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f3b7bc0-513d-4998-b311-b77b7d5a5c71 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:11.170861 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:43:11.182375 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96813649-a1e0-49b8-ba60-8b658a199ab0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:11.183036 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96813649-a1e0-49b8-ba60-8b658a199ab0 None admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:11.183310 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96813649-a1e0-49b8-ba60-8b658a199ab0 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:11.183579 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96813649-a1e0-49b8-ba60-8b658a199ab0 None admin] PATH_INFO: `/v3/users` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:11.184684 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96813649-a1e0-49b8-ba60-8b658a199ab0 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:43:11.185839 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96813649-a1e0-49b8-ba60-8b658a199ab0 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:43:11.209539 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 561/1110] 10.222.0.42 () {60 vars in 1132 bytes} [Sun Aug 28 00:43:11 2022] POST /identity/v3/users => generated 286 bytes in 186 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:43:13.252509 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efb033c5-f51c-4444-9b65-fd2f6f36bd27 None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:13.253142 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efb033c5-f51c-4444-9b65-fd2f6f36bd27 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:13.253142 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efb033c5-f51c-4444-9b65-fd2f6f36bd27 None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:13.254429 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 550/1111] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:43:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:43:13.260398 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cc19b50-7528-4b92-9143-27e9ab43f9fe None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:13.260995 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cc19b50-7528-4b92-9143-27e9ab43f9fe None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:13.260995 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cc19b50-7528-4b92-9143-27e9ab43f9fe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:13.280727 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-9cc19b50-7528-4b92-9143-27e9ab43f9fe None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:43:13.281972 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-9cc19b50-7528-4b92-9143-27e9ab43f9fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:43:13.355228 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 562/1112] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:43:13 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 28 00:43:13.665497 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efb033c5-f51c-4444-9b65-fd2f6f36bd27 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:13.779448 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efb033c5-f51c-4444-9b65-fd2f6f36bd27 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:13.816288 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13a43d70-6c6e-409d-93f9-0f4e5e802520 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:13.816929 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13a43d70-6c6e-409d-93f9-0f4e5e802520 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:13.817220 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13a43d70-6c6e-409d-93f9-0f4e5e802520 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:13.817492 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13a43d70-6c6e-409d-93f9-0f4e5e802520 None admin] PATH_INFO: `/v3/roles/05caf0846a15452b9a6a79fa59c7fa5a` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:13.818892 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13a43d70-6c6e-409d-93f9-0f4e5e802520 None admin] RBAC: Authorizing `identity:get_role(role_id=05caf0846a15452b9a6a79fa59c7fa5a)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:43:13.820264 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13a43d70-6c6e-409d-93f9-0f4e5e802520 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:43:13.821632 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 551/1113] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Aug 28 00:43:13 2022] GET /identity/v3/roles/05caf0846a15452b9a6a79fa59c7fa5a => generated 223 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:43:13.827825 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cc19b50-7528-4b92-9143-27e9ab43f9fe None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:13.935226 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cc19b50-7528-4b92-9143-27e9ab43f9fe None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:13.949180 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:43:13.972572 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9838339-d5bc-495b-a9da-99d5c6ce412f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:13.973246 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9838339-d5bc-495b-a9da-99d5c6ce412f None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:13.973520 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9838339-d5bc-495b-a9da-99d5c6ce412f None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:13.973785 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9838339-d5bc-495b-a9da-99d5c6ce412f None admin] PATH_INFO: `/v3/users/10e4a88d03df4ffcad96543c866d868e` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:13.987968 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9838339-d5bc-495b-a9da-99d5c6ce412f None admin] RBAC: Authorizing `identity:get_user(user_id=10e4a88d03df4ffcad96543c866d868e)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:43:13.989451 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9838339-d5bc-495b-a9da-99d5c6ce412f None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:43:13.991100 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 563/1114] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Aug 28 00:43:13 2022] GET /identity/v3/users/10e4a88d03df4ffcad96543c866d868e => generated 286 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:43:13.997273 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13a43d70-6c6e-409d-93f9-0f4e5e802520 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:14.047948 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:43:14.097977 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13a43d70-6c6e-409d-93f9-0f4e5e802520 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:14.136261 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0c511c3-110e-4e37-be7f-42540b07f60a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:14.136950 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0c511c3-110e-4e37-be7f-42540b07f60a None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:14.137324 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0c511c3-110e-4e37-be7f-42540b07f60a None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:14.137652 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0c511c3-110e-4e37-be7f-42540b07f60a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:14.212537 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0c511c3-110e-4e37-be7f-42540b07f60a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:43:14.213923 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0c511c3-110e-4e37-be7f-42540b07f60a None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:43:14.262952 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:43:14.315720 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 552/1115] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Aug 28 00:43:13 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 319 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 28 00:43:14.322137 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9838339-d5bc-495b-a9da-99d5c6ce412f None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:14.431214 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9838339-d5bc-495b-a9da-99d5c6ce412f None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:14.469239 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97b80b7e-a7cd-4910-8a0c-7b2f7bacdf78 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:14.469787 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b80b7e-a7cd-4910-8a0c-7b2f7bacdf78 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:14.470065 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b80b7e-a7cd-4910-8a0c-7b2f7bacdf78 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:14.470335 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b80b7e-a7cd-4910-8a0c-7b2f7bacdf78 None admin] PATH_INFO: `/v3/projects/a661c03290754db3864357aaaffd5aa3` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:14.471450 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97b80b7e-a7cd-4910-8a0c-7b2f7bacdf78 None admin] RBAC: Authorizing `identity:get_project(project_id=a661c03290754db3864357aaaffd5aa3)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:43:14.472889 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97b80b7e-a7cd-4910-8a0c-7b2f7bacdf78 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:43:14.474555 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 564/1116] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Aug 28 00:43:14 2022] GET /identity/v3/projects/a661c03290754db3864357aaaffd5aa3 => generated 311 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:43:14.481047 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0c511c3-110e-4e37-be7f-42540b07f60a None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:14.586957 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0c511c3-110e-4e37-be7f-42540b07f60a None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:14.621935 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50a5aaca-4ec6-4dd4-8b34-eed275ae4186 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:14.622542 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50a5aaca-4ec6-4dd4-8b34-eed275ae4186 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:14.622822 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50a5aaca-4ec6-4dd4-8b34-eed275ae4186 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:14.623091 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50a5aaca-4ec6-4dd4-8b34-eed275ae4186 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:14.623808 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50a5aaca-4ec6-4dd4-8b34-eed275ae4186 None admin] RBAC: Adding query filter params (role.id=05caf0846a15452b9a6a79fa59c7fa5a, scope.project.id=a661c03290754db3864357aaaffd5aa3, user.id=10e4a88d03df4ffcad96543c866d868e) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:43:14.624117 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50a5aaca-4ec6-4dd4-8b34-eed275ae4186 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:43:14.625592 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50a5aaca-4ec6-4dd4-8b34-eed275ae4186 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:43:14.631254 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 553/1117] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Aug 28 00:43:14 2022] GET /identity/v3/role_assignments?user.id=10e4a88d03df4ffcad96543c866d868e&scope.project.id=a661c03290754db3864357aaaffd5aa3&role.id=05caf0846a15452b9a6a79fa59c7fa5a => generated 259 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:43:16.503954 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18f66697-5b60-4c2f-9f0f-4f89245566c3 None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:16.504500 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18f66697-5b60-4c2f-9f0f-4f89245566c3 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:16.504500 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18f66697-5b60-4c2f-9f0f-4f89245566c3 None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:16.505863 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 565/1118] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:43:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:43:16.512416 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d68b02d9-24a4-4bc4-9880-585bdcabcf15 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:16.512877 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d68b02d9-24a4-4bc4-9880-585bdcabcf15 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:16.513193 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d68b02d9-24a4-4bc4-9880-585bdcabcf15 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:16.534748 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-d68b02d9-24a4-4bc4-9880-585bdcabcf15 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:43:16.536184 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-d68b02d9-24a4-4bc4-9880-585bdcabcf15 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:43:16.548857 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:43:16.627814 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 554/1119] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:43:16 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 28 00:43:16.893137 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18f66697-5b60-4c2f-9f0f-4f89245566c3 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:16.937461 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:43:17.004613 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18f66697-5b60-4c2f-9f0f-4f89245566c3 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:17.043740 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9465d212-3704-4ef6-9730-a2c462cd6350 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:17.044400 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9465d212-3704-4ef6-9730-a2c462cd6350 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:17.044676 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9465d212-3704-4ef6-9730-a2c462cd6350 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:17.044942 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9465d212-3704-4ef6-9730-a2c462cd6350 None admin] PATH_INFO: `/v3/roles/05caf0846a15452b9a6a79fa59c7fa5a` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:17.046458 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9465d212-3704-4ef6-9730-a2c462cd6350 None admin] RBAC: Authorizing `identity:get_role(role_id=05caf0846a15452b9a6a79fa59c7fa5a)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:43:17.047839 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9465d212-3704-4ef6-9730-a2c462cd6350 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:43:17.049230 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 566/1120] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Aug 28 00:43:16 2022] GET /identity/v3/roles/05caf0846a15452b9a6a79fa59c7fa5a => generated 223 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:43:17.055451 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d68b02d9-24a4-4bc4-9880-585bdcabcf15 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:17.162865 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d68b02d9-24a4-4bc4-9880-585bdcabcf15 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:17.198287 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff49e256-cfaa-49b6-b76a-d1379886b5f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:17.198927 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff49e256-cfaa-49b6-b76a-d1379886b5f6 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:17.199204 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff49e256-cfaa-49b6-b76a-d1379886b5f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:17.199474 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff49e256-cfaa-49b6-b76a-d1379886b5f6 None admin] PATH_INFO: `/v3/users/10e4a88d03df4ffcad96543c866d868e` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:17.200684 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff49e256-cfaa-49b6-b76a-d1379886b5f6 None admin] RBAC: Authorizing `identity:get_user(user_id=10e4a88d03df4ffcad96543c866d868e)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:43:17.202152 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff49e256-cfaa-49b6-b76a-d1379886b5f6 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:43:17.203769 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 555/1121] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Aug 28 00:43:17 2022] GET /identity/v3/users/10e4a88d03df4ffcad96543c866d868e => generated 286 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:43:17.210316 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9465d212-3704-4ef6-9730-a2c462cd6350 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:17.321455 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9465d212-3704-4ef6-9730-a2c462cd6350 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:17.330921 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:43:17.362960 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8455f43-fe5a-4235-9190-bffbac39f8b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:17.363656 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8455f43-fe5a-4235-9190-bffbac39f8b4 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:17.363977 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8455f43-fe5a-4235-9190-bffbac39f8b4 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:17.364292 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8455f43-fe5a-4235-9190-bffbac39f8b4 None admin] PATH_INFO: `/v3/projects/a661c03290754db3864357aaaffd5aa3` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:17.365525 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8455f43-fe5a-4235-9190-bffbac39f8b4 None admin] RBAC: Authorizing `identity:get_project(project_id=a661c03290754db3864357aaaffd5aa3)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:43:17.366978 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8455f43-fe5a-4235-9190-bffbac39f8b4 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:43:17.368664 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 567/1122] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Aug 28 00:43:17 2022] GET /identity/v3/projects/a661c03290754db3864357aaaffd5aa3 => generated 311 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:43:17.375016 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff49e256-cfaa-49b6-b76a-d1379886b5f6 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:17.391254 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:43:17.481677 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff49e256-cfaa-49b6-b76a-d1379886b5f6 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:17.518444 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2215c16d-7326-45c5-910c-0c46cf867008 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:17.519434 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2215c16d-7326-45c5-910c-0c46cf867008 None admin] REQUEST_METHOD: `PUT` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:17.519862 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2215c16d-7326-45c5-910c-0c46cf867008 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:17.520275 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2215c16d-7326-45c5-910c-0c46cf867008 None admin] PATH_INFO: `/v3/projects/a661c03290754db3864357aaaffd5aa3/users/10e4a88d03df4ffcad96543c866d868e/roles/05caf0846a15452b9a6a79fa59c7fa5a` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:17.530325 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2215c16d-7326-45c5-910c-0c46cf867008 None admin] RBAC: Authorizing `identity:create_grant(project_id=a661c03290754db3864357aaaffd5aa3, user_id=10e4a88d03df4ffcad96543c866d868e, role_id=05caf0846a15452b9a6a79fa59c7fa5a)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:43:17.531897 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2215c16d-7326-45c5-910c-0c46cf867008 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:43:17.547209 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 556/1123] 10.222.0.42 () {58 vars in 1439 bytes} [Sun Aug 28 00:43:17 2022] PUT /identity/v3/projects/a661c03290754db3864357aaaffd5aa3/users/10e4a88d03df4ffcad96543c866d868e/roles/05caf0846a15452b9a6a79fa59c7fa5a => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:43:19.449509 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4f8e188-0108-4fc3-8e29-77740535cb11 None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:19.450143 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4f8e188-0108-4fc3-8e29-77740535cb11 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:19.450143 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4f8e188-0108-4fc3-8e29-77740535cb11 None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:19.451597 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 568/1124] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:43:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:43:19.458051 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d37cbe3c-2881-41c1-9b45-b31be3b0f01a None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:19.458608 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d37cbe3c-2881-41c1-9b45-b31be3b0f01a None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:19.458719 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d37cbe3c-2881-41c1-9b45-b31be3b0f01a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:19.478596 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-d37cbe3c-2881-41c1-9b45-b31be3b0f01a None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:43:19.479803 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-d37cbe3c-2881-41c1-9b45-b31be3b0f01a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:43:19.565562 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 557/1125] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:43:19 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 28 00:43:19.834350 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4f8e188-0108-4fc3-8e29-77740535cb11 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:19.943354 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4f8e188-0108-4fc3-8e29-77740535cb11 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:19.979641 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f19a1300-f3f2-46a8-8099-d875a9af5d51 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:19.980283 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f19a1300-f3f2-46a8-8099-d875a9af5d51 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:19.980561 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f19a1300-f3f2-46a8-8099-d875a9af5d51 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:19.980830 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f19a1300-f3f2-46a8-8099-d875a9af5d51 None admin] PATH_INFO: `/v3/roles/05caf0846a15452b9a6a79fa59c7fa5a` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:19.982325 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f19a1300-f3f2-46a8-8099-d875a9af5d51 None admin] RBAC: Authorizing `identity:get_role(role_id=05caf0846a15452b9a6a79fa59c7fa5a)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:43:19.983712 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f19a1300-f3f2-46a8-8099-d875a9af5d51 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:43:19.985065 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 569/1126] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Aug 28 00:43:19 2022] GET /identity/v3/roles/05caf0846a15452b9a6a79fa59c7fa5a => generated 223 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:43:19.991257 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d37cbe3c-2881-41c1-9b45-b31be3b0f01a None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:19.993588 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:43:20.103970 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d37cbe3c-2881-41c1-9b45-b31be3b0f01a None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:20.148075 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-276fbdf4-3751-49d7-b88a-07fb51ed9cd2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:20.148873 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-276fbdf4-3751-49d7-b88a-07fb51ed9cd2 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:20.149210 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-276fbdf4-3751-49d7-b88a-07fb51ed9cd2 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:20.149540 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-276fbdf4-3751-49d7-b88a-07fb51ed9cd2 None admin] PATH_INFO: `/v3/users/10e4a88d03df4ffcad96543c866d868e` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:20.151004 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-276fbdf4-3751-49d7-b88a-07fb51ed9cd2 None admin] RBAC: Authorizing `identity:get_user(user_id=10e4a88d03df4ffcad96543c866d868e)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:43:20.152568 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-276fbdf4-3751-49d7-b88a-07fb51ed9cd2 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:43:20.154595 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 558/1127] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Aug 28 00:43:19 2022] GET /identity/v3/users/10e4a88d03df4ffcad96543c866d868e => generated 286 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:43:20.160950 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f19a1300-f3f2-46a8-8099-d875a9af5d51 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:20.208125 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:43:20.273444 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f19a1300-f3f2-46a8-8099-d875a9af5d51 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:20.310757 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ff5a825-fffa-488d-9a37-6b41a558ab42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:20.311361 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ff5a825-fffa-488d-9a37-6b41a558ab42 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:20.311635 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ff5a825-fffa-488d-9a37-6b41a558ab42 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:20.311898 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ff5a825-fffa-488d-9a37-6b41a558ab42 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:20.386321 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ff5a825-fffa-488d-9a37-6b41a558ab42 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:43:20.387888 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ff5a825-fffa-488d-9a37-6b41a558ab42 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:43:20.435986 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:43:20.495589 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 570/1128] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Aug 28 00:43:20 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 335 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 28 00:43:20.502115 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-276fbdf4-3751-49d7-b88a-07fb51ed9cd2 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:20.577222 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:43:20.610088 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-276fbdf4-3751-49d7-b88a-07fb51ed9cd2 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:20.646529 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf654c49-ebc2-4640-bd71-f458f69ccff1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:20.647150 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf654c49-ebc2-4640-bd71-f458f69ccff1 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:20.647476 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf654c49-ebc2-4640-bd71-f458f69ccff1 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:20.647791 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf654c49-ebc2-4640-bd71-f458f69ccff1 None admin] PATH_INFO: `/v3/projects/a661c03290754db3864357aaaffd5aa3` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:20.649147 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf654c49-ebc2-4640-bd71-f458f69ccff1 None admin] RBAC: Authorizing `identity:get_project(project_id=a661c03290754db3864357aaaffd5aa3)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:43:20.650671 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf654c49-ebc2-4640-bd71-f458f69ccff1 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:43:20.652480 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 559/1129] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Aug 28 00:43:20 2022] GET /identity/v3/projects/a661c03290754db3864357aaaffd5aa3 => generated 311 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:43:20.659323 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ff5a825-fffa-488d-9a37-6b41a558ab42 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:20.761965 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ff5a825-fffa-488d-9a37-6b41a558ab42 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:20.801226 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-681cd8b6-a048-4ba1-aa06-30ed5ea2d5a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:20.801842 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-681cd8b6-a048-4ba1-aa06-30ed5ea2d5a0 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:20.802125 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-681cd8b6-a048-4ba1-aa06-30ed5ea2d5a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:20.802401 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-681cd8b6-a048-4ba1-aa06-30ed5ea2d5a0 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:20.803125 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-681cd8b6-a048-4ba1-aa06-30ed5ea2d5a0 None admin] RBAC: Adding query filter params (role.id=05caf0846a15452b9a6a79fa59c7fa5a, scope.project.id=a661c03290754db3864357aaaffd5aa3, user.id=10e4a88d03df4ffcad96543c866d868e) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:43:20.803436 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-681cd8b6-a048-4ba1-aa06-30ed5ea2d5a0 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:43:20.804905 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-681cd8b6-a048-4ba1-aa06-30ed5ea2d5a0 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:43:20.811693 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 571/1130] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Aug 28 00:43:20 2022] GET /identity/v3/role_assignments?user.id=10e4a88d03df4ffcad96543c866d868e&scope.project.id=a661c03290754db3864357aaaffd5aa3&role.id=05caf0846a15452b9a6a79fa59c7fa5a => generated 608 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:43:22.741647 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec66f39-360e-439d-9f5b-3abf461d13b6 None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:22.742228 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec66f39-360e-439d-9f5b-3abf461d13b6 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:22.742228 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec66f39-360e-439d-9f5b-3abf461d13b6 None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:22.743555 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 560/1131] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:43:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:43:22.749577 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64828ee7-d106-44d7-a26c-80112fed1612 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:22.750100 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64828ee7-d106-44d7-a26c-80112fed1612 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:22.750100 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64828ee7-d106-44d7-a26c-80112fed1612 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:22.763455 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:43:22.769570 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-64828ee7-d106-44d7-a26c-80112fed1612 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:43:22.770788 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-64828ee7-d106-44d7-a26c-80112fed1612 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:43:22.842677 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 572/1132] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:43:22 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 28 00:43:23.113733 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dec66f39-360e-439d-9f5b-3abf461d13b6 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:23.230141 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dec66f39-360e-439d-9f5b-3abf461d13b6 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:23.268273 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93d8ad45-3ba7-41c6-b402-2f88f966d291 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:23.268915 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93d8ad45-3ba7-41c6-b402-2f88f966d291 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:23.269283 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93d8ad45-3ba7-41c6-b402-2f88f966d291 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:23.269560 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93d8ad45-3ba7-41c6-b402-2f88f966d291 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:23.270786 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93d8ad45-3ba7-41c6-b402-2f88f966d291 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:43:23.272244 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93d8ad45-3ba7-41c6-b402-2f88f966d291 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:43:23.273881 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 561/1133] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:43:23 2022] GET /identity/v3/domains/default => generated 199 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:43:23.280359 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64828ee7-d106-44d7-a26c-80112fed1612 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:23.386535 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64828ee7-d106-44d7-a26c-80112fed1612 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:23.423564 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc5bb466-ce6c-4c02-ad1f-acae08ee405a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:23.424111 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc5bb466-ce6c-4c02-ad1f-acae08ee405a None admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:23.424388 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc5bb466-ce6c-4c02-ad1f-acae08ee405a None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:23.424654 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc5bb466-ce6c-4c02-ad1f-acae08ee405a None admin] PATH_INFO: `/v3/projects` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:23.425334 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc5bb466-ce6c-4c02-ad1f-acae08ee405a None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:43:23.426784 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc5bb466-ce6c-4c02-ad1f-acae08ee405a None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:43:23.432545 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:43:23.456345 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 573/1134] 10.222.0.42 () {60 vars in 1141 bytes} [Sun Aug 28 00:43:23 2022] POST /identity/v3/projects => generated 311 bytes in 177 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:43:25.419631 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-035b52fa-b089-41a5-967d-75da8c1f0408 None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:25.420187 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-035b52fa-b089-41a5-967d-75da8c1f0408 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:25.420187 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-035b52fa-b089-41a5-967d-75da8c1f0408 None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:25.421561 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 562/1135] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:43:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:43:25.427127 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e731df-a352-49f3-ba7f-5835e0afe826 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:25.427577 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e731df-a352-49f3-ba7f-5835e0afe826 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:25.427669 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e731df-a352-49f3-ba7f-5835e0afe826 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:25.446088 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-97e731df-a352-49f3-ba7f-5835e0afe826 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:43:25.447287 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-97e731df-a352-49f3-ba7f-5835e0afe826 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:43:25.516509 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 574/1136] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:43:25 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 28 00:43:25.785050 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-035b52fa-b089-41a5-967d-75da8c1f0408 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:25.794587 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:43:25.892183 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-035b52fa-b089-41a5-967d-75da8c1f0408 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:25.930830 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e53821c9-53d8-41b9-83b6-b9cfaa6ec6fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:25.931497 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e53821c9-53d8-41b9-83b6-b9cfaa6ec6fc None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:25.931774 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e53821c9-53d8-41b9-83b6-b9cfaa6ec6fc None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:25.932046 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e53821c9-53d8-41b9-83b6-b9cfaa6ec6fc None admin] PATH_INFO: `/v3/domains/default` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:25.933301 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e53821c9-53d8-41b9-83b6-b9cfaa6ec6fc None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:43:25.934812 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e53821c9-53d8-41b9-83b6-b9cfaa6ec6fc None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:43:25.936447 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 563/1137] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:43:25 2022] GET /identity/v3/domains/default => generated 199 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:43:25.942651 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97e731df-a352-49f3-ba7f-5835e0afe826 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:26.051430 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97e731df-a352-49f3-ba7f-5835e0afe826 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:26.069487 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:43:26.088699 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69fca53e-47ab-4453-8e72-2ba762a7f46f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:26.089337 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69fca53e-47ab-4453-8e72-2ba762a7f46f None admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:26.089614 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69fca53e-47ab-4453-8e72-2ba762a7f46f None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:26.089874 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69fca53e-47ab-4453-8e72-2ba762a7f46f None admin] PATH_INFO: `/v3/users` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:26.090580 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69fca53e-47ab-4453-8e72-2ba762a7f46f None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:43:26.092022 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69fca53e-47ab-4453-8e72-2ba762a7f46f None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:43:26.115233 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 575/1138] 10.222.0.42 () {60 vars in 1132 bytes} [Sun Aug 28 00:43:25 2022] POST /identity/v3/users => generated 286 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:43:28.078599 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45c73a42-82a5-4422-b672-e176845aadeb None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:28.079510 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45c73a42-82a5-4422-b672-e176845aadeb None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:28.079510 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45c73a42-82a5-4422-b672-e176845aadeb None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:28.080902 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 564/1139] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:43:28 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:43:28.086844 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6880f9dd-44a8-4ddb-954c-7f8e4a646b6c None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:28.087315 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6880f9dd-44a8-4ddb-954c-7f8e4a646b6c None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:28.087428 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6880f9dd-44a8-4ddb-954c-7f8e4a646b6c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:28.107494 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-6880f9dd-44a8-4ddb-954c-7f8e4a646b6c None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:43:28.108711 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-6880f9dd-44a8-4ddb-954c-7f8e4a646b6c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:43:28.189625 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 576/1140] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:43:28 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 28 00:43:28.463762 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45c73a42-82a5-4422-b672-e176845aadeb None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:28.549179 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:43:28.577472 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45c73a42-82a5-4422-b672-e176845aadeb None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:28.613368 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea441312-bcb9-4a74-9d1d-606b3fe3510d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:28.614063 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea441312-bcb9-4a74-9d1d-606b3fe3510d None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:28.614407 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea441312-bcb9-4a74-9d1d-606b3fe3510d None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:28.614671 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea441312-bcb9-4a74-9d1d-606b3fe3510d None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:28.625337 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea441312-bcb9-4a74-9d1d-606b3fe3510d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 28 00:43:28.630205 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-ea441312-bcb9-4a74-9d1d-606b3fe3510d None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 28 00:43:28.630205 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:43:28.630205 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:43:28.630205 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:43:28.630205 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:43:28.630205 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:43:28.630205 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:43:28.630205 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:43:28.630205 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:43:28.630205 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:43:28.630205 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:43:28.630205 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:43:28.630205 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 28 00:43:28.630205 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 28 00:43:28.630205 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 28 00:43:28.630205 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 28 00:43:28.632030 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 28 00:43:28.632030 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 28 00:43:28.632030 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 28 00:43:28.632030 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 28 00:43:28.632030 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:43:28.632030 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:43:28.632030 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:43:28.632030 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:43:28.632030 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:43:28.632030 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:43:28.632030 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:43:28.632030 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:43:28.632030 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:43:28.632030 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:43:28.632030 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:43:28.632030 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:43:28.633317 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:43:28.633317 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:43:28.633317 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:43:28.633317 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:43:28.633317 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 28 00:43:28.633317 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 28 00:43:28.633317 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 28 00:43:28.633317 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 28 00:43:28.633317 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 28 00:43:28.633317 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 28 00:43:28.633317 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 28 00:43:28.633317 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:43:28.633317 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 565/1141] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Aug 28 00:43:28 2022] GET /identity/v3/roles/admin => generated 83 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:43:28.638554 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6880f9dd-44a8-4ddb-954c-7f8e4a646b6c None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:28.743096 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6880f9dd-44a8-4ddb-954c-7f8e4a646b6c None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:28.745422 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:43:28.779075 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1308d451-5ba4-46c1-8735-daef6b9fabca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:28.779724 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1308d451-5ba4-46c1-8735-daef6b9fabca None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:28.779998 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1308d451-5ba4-46c1-8735-daef6b9fabca None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:28.780263 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1308d451-5ba4-46c1-8735-daef6b9fabca None admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:28.780944 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1308d451-5ba4-46c1-8735-daef6b9fabca None admin] RBAC: Adding query filter params (name=admin) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:43:28.781270 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1308d451-5ba4-46c1-8735-daef6b9fabca None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:43:28.782657 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1308d451-5ba4-46c1-8735-daef6b9fabca None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:43:28.791761 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 577/1142] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Aug 28 00:43:28 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:43:28.797961 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea441312-bcb9-4a74-9d1d-606b3fe3510d None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:28.923777 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea441312-bcb9-4a74-9d1d-606b3fe3510d None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:28.964106 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6758f6a5-6fd7-4a2e-b1f6-306f89aed8ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:28.964833 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6758f6a5-6fd7-4a2e-b1f6-306f89aed8ff None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:28.965304 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6758f6a5-6fd7-4a2e-b1f6-306f89aed8ff None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:28.965577 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6758f6a5-6fd7-4a2e-b1f6-306f89aed8ff None admin] PATH_INFO: `/v3/users/88f46d08d637454b864c0687c4366435` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:28.967691 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:43:28.982366 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6758f6a5-6fd7-4a2e-b1f6-306f89aed8ff None admin] RBAC: Authorizing `identity:get_user(user_id=88f46d08d637454b864c0687c4366435)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:43:28.983843 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6758f6a5-6fd7-4a2e-b1f6-306f89aed8ff None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:43:28.985775 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 566/1143] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Aug 28 00:43:28 2022] GET /identity/v3/users/88f46d08d637454b864c0687c4366435 => generated 286 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:43:28.991906 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1308d451-5ba4-46c1-8735-daef6b9fabca None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:29.095085 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1308d451-5ba4-46c1-8735-daef6b9fabca None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:29.150953 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54d418fe-06e4-429c-a6c6-2896916e8b84 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:29.151550 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54d418fe-06e4-429c-a6c6-2896916e8b84 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:29.151828 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54d418fe-06e4-429c-a6c6-2896916e8b84 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:29.152094 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54d418fe-06e4-429c-a6c6-2896916e8b84 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:29.186762 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:43:29.221894 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54d418fe-06e4-429c-a6c6-2896916e8b84 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:43:29.223346 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54d418fe-06e4-429c-a6c6-2896916e8b84 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:43:29.321597 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 578/1144] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Aug 28 00:43:28 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 330 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 28 00:43:29.328841 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6758f6a5-6fd7-4a2e-b1f6-306f89aed8ff None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:29.436949 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6758f6a5-6fd7-4a2e-b1f6-306f89aed8ff None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:29.474039 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e62fb77-ca3d-4d66-8d9d-7a15994eed9a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:29.474697 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e62fb77-ca3d-4d66-8d9d-7a15994eed9a None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:29.474973 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e62fb77-ca3d-4d66-8d9d-7a15994eed9a None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:29.475239 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e62fb77-ca3d-4d66-8d9d-7a15994eed9a None admin] PATH_INFO: `/v3/projects/7a3866d4f56249c491753241c583e06c` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:29.476473 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e62fb77-ca3d-4d66-8d9d-7a15994eed9a None admin] RBAC: Authorizing `identity:get_project(project_id=7a3866d4f56249c491753241c583e06c)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:43:29.477992 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e62fb77-ca3d-4d66-8d9d-7a15994eed9a None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:43:29.479778 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 567/1145] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Aug 28 00:43:29 2022] GET /identity/v3/projects/7a3866d4f56249c491753241c583e06c => generated 311 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:43:29.486682 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54d418fe-06e4-429c-a6c6-2896916e8b84 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:29.564837 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:43:29.606008 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54d418fe-06e4-429c-a6c6-2896916e8b84 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:29.642458 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-702ee5eb-582e-481a-8cd5-87c87a3f812f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:29.643066 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-702ee5eb-582e-481a-8cd5-87c87a3f812f None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:29.643342 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-702ee5eb-582e-481a-8cd5-87c87a3f812f None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:29.643608 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-702ee5eb-582e-481a-8cd5-87c87a3f812f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:29.644334 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-702ee5eb-582e-481a-8cd5-87c87a3f812f None admin] RBAC: Adding query filter params (role.id=f08d6043261747e3b4763c744eef7dfb, scope.project.id=7a3866d4f56249c491753241c583e06c, user.id=88f46d08d637454b864c0687c4366435) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:43:29.644634 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-702ee5eb-582e-481a-8cd5-87c87a3f812f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:43:29.646103 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-702ee5eb-582e-481a-8cd5-87c87a3f812f None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:43:29.654230 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 579/1146] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Aug 28 00:43:29 2022] GET /identity/v3/role_assignments?user.id=88f46d08d637454b864c0687c4366435&scope.project.id=7a3866d4f56249c491753241c583e06c&role.id=f08d6043261747e3b4763c744eef7dfb => generated 259 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:43:31.544564 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a3072cb-83ce-4796-998c-9d4327fa319d None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:31.545157 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a3072cb-83ce-4796-998c-9d4327fa319d None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:31.545157 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a3072cb-83ce-4796-998c-9d4327fa319d None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:31.546452 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 568/1147] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:43:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:43:31.552057 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce52402d-5638-4bc6-8f5a-4993696749e8 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:31.552486 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce52402d-5638-4bc6-8f5a-4993696749e8 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:31.552601 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce52402d-5638-4bc6-8f5a-4993696749e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:31.571878 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-ce52402d-5638-4bc6-8f5a-4993696749e8 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:43:31.573131 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-ce52402d-5638-4bc6-8f5a-4993696749e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:43:31.647016 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 580/1148] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:43:31 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 28 00:43:31.909828 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a3072cb-83ce-4796-998c-9d4327fa319d None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:31.972579 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:43:32.015612 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a3072cb-83ce-4796-998c-9d4327fa319d None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:32.051236 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29212dec-fe22-4861-bca0-6a4eae0032a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:32.051864 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29212dec-fe22-4861-bca0-6a4eae0032a9 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:32.052141 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29212dec-fe22-4861-bca0-6a4eae0032a9 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:32.052406 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29212dec-fe22-4861-bca0-6a4eae0032a9 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:32.063156 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29212dec-fe22-4861-bca0-6a4eae0032a9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 28 00:43:32.068049 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-29212dec-fe22-4861-bca0-6a4eae0032a9 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 28 00:43:32.068049 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:43:32.068049 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:43:32.068049 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:43:32.068049 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:43:32.068049 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:43:32.068049 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:43:32.068049 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:43:32.068049 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:43:32.068049 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:43:32.068049 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:43:32.068049 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:43:32.068049 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 28 00:43:32.068049 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 28 00:43:32.068049 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 28 00:43:32.068049 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 28 00:43:32.069168 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 28 00:43:32.069168 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 28 00:43:32.069168 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 28 00:43:32.069168 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 28 00:43:32.069168 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:43:32.069168 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:43:32.069168 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:43:32.069168 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:43:32.069168 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:43:32.069168 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:43:32.069168 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:43:32.069168 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:43:32.069168 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:43:32.069168 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:43:32.069168 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:43:32.069168 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:43:32.070200 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:43:32.070200 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:43:32.070200 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:43:32.070200 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:43:32.070200 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 28 00:43:32.070200 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 28 00:43:32.070200 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 28 00:43:32.070200 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 28 00:43:32.070200 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 28 00:43:32.070200 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 28 00:43:32.070200 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 28 00:43:32.070200 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:43:32.070200 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 569/1149] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Aug 28 00:43:31 2022] GET /identity/v3/roles/admin => generated 83 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:43:32.075573 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce52402d-5638-4bc6-8f5a-4993696749e8 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:32.139898 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:43:32.183428 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce52402d-5638-4bc6-8f5a-4993696749e8 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:32.225449 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cb1d1da-98dc-4f05-a292-ae7efa1e69e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:32.226083 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cb1d1da-98dc-4f05-a292-ae7efa1e69e2 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:32.226361 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cb1d1da-98dc-4f05-a292-ae7efa1e69e2 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:32.226652 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cb1d1da-98dc-4f05-a292-ae7efa1e69e2 None admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:32.227340 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cb1d1da-98dc-4f05-a292-ae7efa1e69e2 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:43:32.227640 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cb1d1da-98dc-4f05-a292-ae7efa1e69e2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:43:32.229103 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cb1d1da-98dc-4f05-a292-ae7efa1e69e2 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:43:32.239232 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 581/1150] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Aug 28 00:43:32 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:43:32.245879 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29212dec-fe22-4861-bca0-6a4eae0032a9 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:32.353956 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29212dec-fe22-4861-bca0-6a4eae0032a9 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:32.374629 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:43:32.390065 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfdea8dd-49cd-483f-a471-eb66e3f0f9da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:32.390721 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfdea8dd-49cd-483f-a471-eb66e3f0f9da None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:32.390997 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfdea8dd-49cd-483f-a471-eb66e3f0f9da None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:32.391247 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfdea8dd-49cd-483f-a471-eb66e3f0f9da None admin] PATH_INFO: `/v3/users/88f46d08d637454b864c0687c4366435` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:32.392405 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfdea8dd-49cd-483f-a471-eb66e3f0f9da None admin] RBAC: Authorizing `identity:get_user(user_id=88f46d08d637454b864c0687c4366435)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:43:32.393883 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfdea8dd-49cd-483f-a471-eb66e3f0f9da None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:43:32.395472 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 570/1151] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Aug 28 00:43:32 2022] GET /identity/v3/users/88f46d08d637454b864c0687c4366435 => generated 286 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:43:32.401536 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cb1d1da-98dc-4f05-a292-ae7efa1e69e2 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:32.504589 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cb1d1da-98dc-4f05-a292-ae7efa1e69e2 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:32.527277 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:43:32.540133 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-378d371f-d0a3-4cfa-ac49-7c231a085504 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:32.540750 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-378d371f-d0a3-4cfa-ac49-7c231a085504 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:32.541048 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-378d371f-d0a3-4cfa-ac49-7c231a085504 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:32.541317 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-378d371f-d0a3-4cfa-ac49-7c231a085504 None admin] PATH_INFO: `/v3/projects/7a3866d4f56249c491753241c583e06c` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:32.542420 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-378d371f-d0a3-4cfa-ac49-7c231a085504 None admin] RBAC: Authorizing `identity:get_project(project_id=7a3866d4f56249c491753241c583e06c)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:43:32.543944 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-378d371f-d0a3-4cfa-ac49-7c231a085504 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:43:32.545606 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 582/1152] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Aug 28 00:43:32 2022] GET /identity/v3/projects/7a3866d4f56249c491753241c583e06c => generated 311 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:43:32.551830 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfdea8dd-49cd-483f-a471-eb66e3f0f9da None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:32.650746 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfdea8dd-49cd-483f-a471-eb66e3f0f9da None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:32.684168 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-774502ad-390d-47b9-857e-8cae5691a45c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:32.684819 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-774502ad-390d-47b9-857e-8cae5691a45c None admin] REQUEST_METHOD: `PUT` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:32.685125 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-774502ad-390d-47b9-857e-8cae5691a45c None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:32.685400 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-774502ad-390d-47b9-857e-8cae5691a45c None admin] PATH_INFO: `/v3/projects/7a3866d4f56249c491753241c583e06c/users/88f46d08d637454b864c0687c4366435/roles/f08d6043261747e3b4763c744eef7dfb` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:32.687419 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-774502ad-390d-47b9-857e-8cae5691a45c None admin] RBAC: Authorizing `identity:create_grant(project_id=7a3866d4f56249c491753241c583e06c, user_id=88f46d08d637454b864c0687c4366435, role_id=f08d6043261747e3b4763c744eef7dfb)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:43:32.688880 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-774502ad-390d-47b9-857e-8cae5691a45c None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:43:32.703291 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 571/1153] 10.222.0.42 () {58 vars in 1439 bytes} [Sun Aug 28 00:43:32 2022] PUT /identity/v3/projects/7a3866d4f56249c491753241c583e06c/users/88f46d08d637454b864c0687c4366435/roles/f08d6043261747e3b4763c744eef7dfb => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:43:34.620861 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab2301c3-98bf-4269-8423-c1eb1442cf00 None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:34.621450 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab2301c3-98bf-4269-8423-c1eb1442cf00 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:34.621450 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab2301c3-98bf-4269-8423-c1eb1442cf00 None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:34.622810 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 583/1154] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:43:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:43:34.628292 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ccd308e-82b2-40e8-9f04-c9014180726c None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:34.628739 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ccd308e-82b2-40e8-9f04-c9014180726c None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:34.628849 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ccd308e-82b2-40e8-9f04-c9014180726c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:34.648273 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-5ccd308e-82b2-40e8-9f04-c9014180726c None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:43:34.649511 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-5ccd308e-82b2-40e8-9f04-c9014180726c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:43:34.696051 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:43:34.722942 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 572/1155] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:43:34 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 28 00:43:35.017796 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab2301c3-98bf-4269-8423-c1eb1442cf00 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:35.126982 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab2301c3-98bf-4269-8423-c1eb1442cf00 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:35.164216 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ee2ceae-60f4-4e11-875d-7e86410cb064 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:35.165419 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ee2ceae-60f4-4e11-875d-7e86410cb064 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:35.165746 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ee2ceae-60f4-4e11-875d-7e86410cb064 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:35.165972 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ee2ceae-60f4-4e11-875d-7e86410cb064 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:35.179897 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ee2ceae-60f4-4e11-875d-7e86410cb064 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 28 00:43:35.185712 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-1ee2ceae-60f4-4e11-875d-7e86410cb064 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 28 00:43:35.185712 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:43:35.185712 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:43:35.185712 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:43:35.185712 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:43:35.185712 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:43:35.185712 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:43:35.185712 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:43:35.185712 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:43:35.185712 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:43:35.185712 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:43:35.185712 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:43:35.185712 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 28 00:43:35.185712 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 28 00:43:35.185712 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 28 00:43:35.185712 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 28 00:43:35.186842 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 28 00:43:35.186842 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 28 00:43:35.186842 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 28 00:43:35.186842 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 28 00:43:35.186842 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:43:35.186842 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:43:35.186842 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:43:35.186842 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:43:35.186842 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:43:35.186842 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:43:35.186842 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:43:35.186842 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:43:35.186842 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:43:35.186842 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:43:35.186842 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:43:35.186842 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:43:35.188071 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:43:35.188071 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:43:35.188071 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:43:35.188071 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:43:35.188071 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 28 00:43:35.188071 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 28 00:43:35.188071 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 28 00:43:35.188071 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 28 00:43:35.188071 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 28 00:43:35.188071 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 28 00:43:35.188071 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 28 00:43:35.188071 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:43:35.188071 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 584/1156] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Aug 28 00:43:35 2022] GET /identity/v3/roles/admin => generated 83 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:43:35.193164 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ccd308e-82b2-40e8-9f04-c9014180726c None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:35.311565 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ccd308e-82b2-40e8-9f04-c9014180726c None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:35.346782 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a55db91f-1ce2-475e-b3e8-3dd7bd508707 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:35.347389 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a55db91f-1ce2-475e-b3e8-3dd7bd508707 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:35.347778 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a55db91f-1ce2-475e-b3e8-3dd7bd508707 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:35.348052 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a55db91f-1ce2-475e-b3e8-3dd7bd508707 None admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:35.348708 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a55db91f-1ce2-475e-b3e8-3dd7bd508707 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:43:35.349001 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a55db91f-1ce2-475e-b3e8-3dd7bd508707 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:43:35.350402 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a55db91f-1ce2-475e-b3e8-3dd7bd508707 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:43:35.358826 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 573/1157] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Aug 28 00:43:35 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:43:35.365020 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ee2ceae-60f4-4e11-875d-7e86410cb064 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:35.419267 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:43:35.469987 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ee2ceae-60f4-4e11-875d-7e86410cb064 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:35.503749 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19413abd-8504-4d43-9002-7abeada412f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:35.504386 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19413abd-8504-4d43-9002-7abeada412f2 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:35.504670 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19413abd-8504-4d43-9002-7abeada412f2 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:35.504945 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19413abd-8504-4d43-9002-7abeada412f2 None admin] PATH_INFO: `/v3/users/88f46d08d637454b864c0687c4366435` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:35.506104 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19413abd-8504-4d43-9002-7abeada412f2 None admin] RBAC: Authorizing `identity:get_user(user_id=88f46d08d637454b864c0687c4366435)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:43:35.507547 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19413abd-8504-4d43-9002-7abeada412f2 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:43:35.509173 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 585/1158] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Aug 28 00:43:35 2022] GET /identity/v3/users/88f46d08d637454b864c0687c4366435 => generated 286 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:43:35.515580 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a55db91f-1ce2-475e-b3e8-3dd7bd508707 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:35.560888 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:43:35.620839 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a55db91f-1ce2-475e-b3e8-3dd7bd508707 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:35.656635 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb7e9490-117b-4a2f-a5f8-bc2539d1154e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:35.657402 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb7e9490-117b-4a2f-a5f8-bc2539d1154e None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:35.657527 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb7e9490-117b-4a2f-a5f8-bc2539d1154e None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:35.657823 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb7e9490-117b-4a2f-a5f8-bc2539d1154e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:35.732416 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb7e9490-117b-4a2f-a5f8-bc2539d1154e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:43:35.733939 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb7e9490-117b-4a2f-a5f8-bc2539d1154e None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:43:35.777321 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:43:35.840986 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 574/1159] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Aug 28 00:43:35 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 326 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 28 00:43:35.847433 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19413abd-8504-4d43-9002-7abeada412f2 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:35.952636 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19413abd-8504-4d43-9002-7abeada412f2 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:35.970288 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:43:35.988648 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5221fc1b-1dce-4d21-a487-07a2a1bf9c77 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:35.989312 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5221fc1b-1dce-4d21-a487-07a2a1bf9c77 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:35.989590 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5221fc1b-1dce-4d21-a487-07a2a1bf9c77 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:35.989861 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5221fc1b-1dce-4d21-a487-07a2a1bf9c77 None admin] PATH_INFO: `/v3/projects/7a3866d4f56249c491753241c583e06c` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:35.991019 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5221fc1b-1dce-4d21-a487-07a2a1bf9c77 None admin] RBAC: Authorizing `identity:get_project(project_id=7a3866d4f56249c491753241c583e06c)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:43:35.992462 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5221fc1b-1dce-4d21-a487-07a2a1bf9c77 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:43:35.994122 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 586/1160] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Aug 28 00:43:35 2022] GET /identity/v3/projects/7a3866d4f56249c491753241c583e06c => generated 311 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:43:36.000570 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb7e9490-117b-4a2f-a5f8-bc2539d1154e None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:36.101390 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb7e9490-117b-4a2f-a5f8-bc2539d1154e None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:36.139265 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-464373bd-3809-493f-aad8-3730be4952e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:36.139874 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-464373bd-3809-493f-aad8-3730be4952e9 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:36.140151 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-464373bd-3809-493f-aad8-3730be4952e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:36.140416 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-464373bd-3809-493f-aad8-3730be4952e9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:36.141189 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-464373bd-3809-493f-aad8-3730be4952e9 None admin] RBAC: Adding query filter params (role.id=f08d6043261747e3b4763c744eef7dfb, scope.project.id=7a3866d4f56249c491753241c583e06c, user.id=88f46d08d637454b864c0687c4366435) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:43:36.141493 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-464373bd-3809-493f-aad8-3730be4952e9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:43:36.142971 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-464373bd-3809-493f-aad8-3730be4952e9 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:43:36.150021 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 575/1161] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Aug 28 00:43:35 2022] GET /identity/v3/role_assignments?user.id=88f46d08d637454b864c0687c4366435&scope.project.id=7a3866d4f56249c491753241c583e06c&role.id=f08d6043261747e3b4763c744eef7dfb => generated 608 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:43:38.087182 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a02f6f-7405-4899-ba5f-42f055187e8d None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:38.087769 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a02f6f-7405-4899-ba5f-42f055187e8d None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:38.087769 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a02f6f-7405-4899-ba5f-42f055187e8d None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:38.089068 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 587/1162] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:43:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:43:38.094652 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60161c31-495a-4447-93be-d55bca0fc8e0 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:38.095203 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60161c31-495a-4447-93be-d55bca0fc8e0 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:38.095203 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60161c31-495a-4447-93be-d55bca0fc8e0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:38.103061 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:43:38.114827 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-60161c31-495a-4447-93be-d55bca0fc8e0 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:43:38.116051 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-60161c31-495a-4447-93be-d55bca0fc8e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:43:38.188602 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 576/1163] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:43:38 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 28 00:43:38.458368 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5a02f6f-7405-4899-ba5f-42f055187e8d None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:38.570158 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5a02f6f-7405-4899-ba5f-42f055187e8d None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:38.610088 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47508461-00ea-4385-80c5-5ffb8b49c6fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:38.610706 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47508461-00ea-4385-80c5-5ffb8b49c6fa None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:38.610976 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47508461-00ea-4385-80c5-5ffb8b49c6fa None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:38.611301 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47508461-00ea-4385-80c5-5ffb8b49c6fa None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:38.666810 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:43:38.680701 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47508461-00ea-4385-80c5-5ffb8b49c6fa None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:43:38.682239 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47508461-00ea-4385-80c5-5ffb8b49c6fa None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:43:38.791507 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 588/1164] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Aug 28 00:43:38 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 334 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 28 00:43:38.797920 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60161c31-495a-4447-93be-d55bca0fc8e0 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:38.902596 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60161c31-495a-4447-93be-d55bca0fc8e0 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:38.938375 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3855efd1-9f38-433a-a322-e697b6703e70 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:38.939124 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3855efd1-9f38-433a-a322-e697b6703e70 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:38.939504 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3855efd1-9f38-433a-a322-e697b6703e70 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:38.939860 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3855efd1-9f38-433a-a322-e697b6703e70 None admin] PATH_INFO: `/v3/domains/swift_test` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:38.941990 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:43:38.948201 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3855efd1-9f38-433a-a322-e697b6703e70 None admin] RBAC: Authorizing `identity:get_domain(domain_id=swift_test)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:43:38.950020 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3855efd1-9f38-433a-a322-e697b6703e70 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:43:38.962298 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-3855efd1-9f38-433a-a322-e697b6703e70 None admin] Could not find domain: swift_test.: keystone.exception.DomainNotFound: Could not find domain: swift_test. Aug 28 00:43:38.962298 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:43:38.962298 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 712, in get_domain Aug 28 00:43:38.962298 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application project = self.driver.get_project(domain_id) Aug 28 00:43:38.962298 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 28 00:43:38.962298 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 28 00:43:38.962298 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 28 00:43:38.962298 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 28 00:43:38.962298 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: swift_test. Aug 28 00:43:38.962298 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:43:38.962298 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 28 00:43:38.962298 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:43:38.962298 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:43:38.962298 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:43:38.962298 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:43:38.962298 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:43:38.962298 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:43:38.963613 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:43:38.963613 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:43:38.963613 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:43:38.963613 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:43:38.963613 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:43:38.963613 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:43:38.963613 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 28 00:43:38.963613 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 28 00:43:38.963613 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 28 00:43:38.963613 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 28 00:43:38.963613 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:43:38.963613 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:43:38.963613 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:43:38.963613 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:43:38.963613 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:43:38.963613 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:43:38.964781 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:43:38.964781 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:43:38.964781 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:43:38.964781 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:43:38.964781 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:43:38.964781 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:43:38.964781 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:43:38.964781 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:43:38.964781 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:43:38.964781 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:43:38.964781 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 718, in get_domain Aug 28 00:43:38.964781 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 28 00:43:38.964781 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: swift_test. Aug 28 00:43:38.964781 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:43:38.964781 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 577/1165] 10.222.0.42 () {56 vars in 1117 bytes} [Sun Aug 28 00:43:38 2022] GET /identity/v3/domains/swift_test => generated 90 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:43:38.970681 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47508461-00ea-4385-80c5-5ffb8b49c6fa None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:39.061711 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:43:39.080901 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47508461-00ea-4385-80c5-5ffb8b49c6fa None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:39.123797 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae4c7e94-555c-4348-bbf7-2b876e142a29 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:39.124763 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae4c7e94-555c-4348-bbf7-2b876e142a29 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:39.125189 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae4c7e94-555c-4348-bbf7-2b876e142a29 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:39.125554 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae4c7e94-555c-4348-bbf7-2b876e142a29 None admin] PATH_INFO: `/v3/domains` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:39.126321 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae4c7e94-555c-4348-bbf7-2b876e142a29 None admin] RBAC: Adding query filter params (name=swift_test) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:43:39.126704 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae4c7e94-555c-4348-bbf7-2b876e142a29 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:43:39.128202 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae4c7e94-555c-4348-bbf7-2b876e142a29 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:43:39.136493 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 589/1166] 10.222.0.42 () {56 vars in 1115 bytes} [Sun Aug 28 00:43:38 2022] GET /identity/v3/domains?name=swift_test => generated 125 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:43:41.021326 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78197379-60eb-4d81-8434-06da498b767f None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:41.021904 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78197379-60eb-4d81-8434-06da498b767f None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:41.021904 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78197379-60eb-4d81-8434-06da498b767f None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:41.023208 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 578/1167] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:43:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:43:41.028756 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb11f62-6893-4bce-9d1c-bf4ac989bb6f None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:41.029213 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb11f62-6893-4bce-9d1c-bf4ac989bb6f None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:41.029327 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb11f62-6893-4bce-9d1c-bf4ac989bb6f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:41.048438 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-1eb11f62-6893-4bce-9d1c-bf4ac989bb6f None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:43:41.049698 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-1eb11f62-6893-4bce-9d1c-bf4ac989bb6f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:43:41.120293 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 590/1168] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:43:41 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 28 00:43:41.387244 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78197379-60eb-4d81-8434-06da498b767f None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:41.495202 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78197379-60eb-4d81-8434-06da498b767f None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:41.532043 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d09953a4-9340-45b9-9d8e-638af4141d5a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:41.532695 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d09953a4-9340-45b9-9d8e-638af4141d5a None admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:41.532983 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d09953a4-9340-45b9-9d8e-638af4141d5a None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:41.533264 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d09953a4-9340-45b9-9d8e-638af4141d5a None admin] PATH_INFO: `/v3/domains` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:41.533951 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d09953a4-9340-45b9-9d8e-638af4141d5a None admin] RBAC: Authorizing `identity:create_domain()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:43:41.535373 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d09953a4-9340-45b9-9d8e-638af4141d5a None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:43:41.555718 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 579/1169] 10.222.0.42 () {60 vars in 1138 bytes} [Sun Aug 28 00:43:41 2022] POST /identity/v3/domains => generated 267 bytes in 168 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:43:43.511259 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-810837e6-854c-41c6-966e-565b83952f42 None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:43.511983 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-810837e6-854c-41c6-966e-565b83952f42 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:43.511983 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-810837e6-854c-41c6-966e-565b83952f42 None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:43.513221 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 591/1170] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:43:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:43:43.518969 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99795e2f-19f5-418f-b696-15d2cc731d6e None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:43.519497 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99795e2f-19f5-418f-b696-15d2cc731d6e None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:43.519497 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99795e2f-19f5-418f-b696-15d2cc731d6e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:43.537858 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-99795e2f-19f5-418f-b696-15d2cc731d6e None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:43:43.539103 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-99795e2f-19f5-418f-b696-15d2cc731d6e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:43:43.590530 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:43:43.609178 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 580/1171] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:43:43 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 28 00:43:43.879243 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-810837e6-854c-41c6-966e-565b83952f42 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:43.967185 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:43:43.987436 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-810837e6-854c-41c6-966e-565b83952f42 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:44.022248 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f2e2af5-7fef-45b1-8c83-3238506b0079 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:44.022875 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f2e2af5-7fef-45b1-8c83-3238506b0079 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:44.023151 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f2e2af5-7fef-45b1-8c83-3238506b0079 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:44.023416 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f2e2af5-7fef-45b1-8c83-3238506b0079 None admin] PATH_INFO: `/v3/domains/f774a0573fe248f5a5a29cec48dd71f6` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:44.034346 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f2e2af5-7fef-45b1-8c83-3238506b0079 None admin] RBAC: Authorizing `identity:get_domain(domain_id=f774a0573fe248f5a5a29cec48dd71f6)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:43:44.035802 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f2e2af5-7fef-45b1-8c83-3238506b0079 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:43:44.037425 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 592/1172] 10.222.0.42 () {56 vars in 1183 bytes} [Sun Aug 28 00:43:43 2022] GET /identity/v3/domains/f774a0573fe248f5a5a29cec48dd71f6 => generated 267 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 28 00:43:44.043648 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99795e2f-19f5-418f-b696-15d2cc731d6e None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:44.147966 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99795e2f-19f5-418f-b696-15d2cc731d6e None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:44.183509 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a33d4119-d671-4ef5-a91d-e38a8db86fb9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:44.184125 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a33d4119-d671-4ef5-a91d-e38a8db86fb9 None admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:44.184400 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a33d4119-d671-4ef5-a91d-e38a8db86fb9 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:44.184670 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a33d4119-d671-4ef5-a91d-e38a8db86fb9 None admin] PATH_INFO: `/v3/projects` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:44.185385 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a33d4119-d671-4ef5-a91d-e38a8db86fb9 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:43:44.186840 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a33d4119-d671-4ef5-a91d-e38a8db86fb9 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:43:44.216282 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 581/1173] 10.222.0.42 () {60 vars in 1141 bytes} [Sun Aug 28 00:43:44 2022] POST /identity/v3/projects => generated 361 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:43:46.118712 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e33d7319-eadd-4f2f-aa2f-ec0c4733e4b6 None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:46.119177 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e33d7319-eadd-4f2f-aa2f-ec0c4733e4b6 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:46.119276 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e33d7319-eadd-4f2f-aa2f-ec0c4733e4b6 None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:46.120598 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 593/1174] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:43:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:43:46.126242 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f734a0e0-4dee-49a5-839a-d9b0eb6b3410 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:46.126726 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f734a0e0-4dee-49a5-839a-d9b0eb6b3410 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:46.126726 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f734a0e0-4dee-49a5-839a-d9b0eb6b3410 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:46.145299 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-f734a0e0-4dee-49a5-839a-d9b0eb6b3410 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:43:46.146497 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-f734a0e0-4dee-49a5-839a-d9b0eb6b3410 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:43:46.176867 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:43:46.221235 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 582/1175] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:43:46 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 28 00:43:46.493150 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e33d7319-eadd-4f2f-aa2f-ec0c4733e4b6 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:46.598353 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e33d7319-eadd-4f2f-aa2f-ec0c4733e4b6 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:46.632766 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4698e9a3-82c9-43cd-ac15-aae131e0948d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:46.633422 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4698e9a3-82c9-43cd-ac15-aae131e0948d None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:46.633699 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4698e9a3-82c9-43cd-ac15-aae131e0948d None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:46.633968 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4698e9a3-82c9-43cd-ac15-aae131e0948d None admin] PATH_INFO: `/v3/domains/f774a0573fe248f5a5a29cec48dd71f6` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:46.635133 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4698e9a3-82c9-43cd-ac15-aae131e0948d None admin] RBAC: Authorizing `identity:get_domain(domain_id=f774a0573fe248f5a5a29cec48dd71f6)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:43:46.636598 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4698e9a3-82c9-43cd-ac15-aae131e0948d None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:43:46.638223 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 594/1176] 10.222.0.42 () {56 vars in 1183 bytes} [Sun Aug 28 00:43:46 2022] GET /identity/v3/domains/f774a0573fe248f5a5a29cec48dd71f6 => generated 267 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:43:46.644290 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f734a0e0-4dee-49a5-839a-d9b0eb6b3410 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:46.748116 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f734a0e0-4dee-49a5-839a-d9b0eb6b3410 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:46.783381 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e650363c-7d3c-4da4-90d0-a7911d788db9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:46.783988 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e650363c-7d3c-4da4-90d0-a7911d788db9 None admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:46.784268 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e650363c-7d3c-4da4-90d0-a7911d788db9 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:46.784531 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e650363c-7d3c-4da4-90d0-a7911d788db9 None admin] PATH_INFO: `/v3/users` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:46.785211 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e650363c-7d3c-4da4-90d0-a7911d788db9 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:43:46.786640 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e650363c-7d3c-4da4-90d0-a7911d788db9 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:43:46.809188 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 583/1177] 10.222.0.42 () {60 vars in 1132 bytes} [Sun Aug 28 00:43:46 2022] POST /identity/v3/users => generated 311 bytes in 165 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:43:48.766802 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0530a703-8107-4040-98ed-fea9e5b1ca89 None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:48.767432 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0530a703-8107-4040-98ed-fea9e5b1ca89 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:48.767432 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0530a703-8107-4040-98ed-fea9e5b1ca89 None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:48.768701 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 595/1178] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:43:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:43:48.774020 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d4be708-f042-42c5-8c43-6dab2a894040 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:48.774460 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d4be708-f042-42c5-8c43-6dab2a894040 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:48.774565 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d4be708-f042-42c5-8c43-6dab2a894040 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:48.792772 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-2d4be708-f042-42c5-8c43-6dab2a894040 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:43:48.794135 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-2d4be708-f042-42c5-8c43-6dab2a894040 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:43:48.811017 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:43:48.863212 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 584/1179] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:43:48 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 28 00:43:49.123468 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0530a703-8107-4040-98ed-fea9e5b1ca89 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:49.135617 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:43:49.231004 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0530a703-8107-4040-98ed-fea9e5b1ca89 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:49.264993 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a392c29-d911-469e-a906-5c30f38ecd59 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:49.265624 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a392c29-d911-469e-a906-5c30f38ecd59 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:49.265868 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a392c29-d911-469e-a906-5c30f38ecd59 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:49.266152 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a392c29-d911-469e-a906-5c30f38ecd59 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:49.277111 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a392c29-d911-469e-a906-5c30f38ecd59 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 28 00:43:49.281871 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-1a392c29-d911-469e-a906-5c30f38ecd59 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 28 00:43:49.281871 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:43:49.281871 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:43:49.281871 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:43:49.281871 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:43:49.281871 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:43:49.281871 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:43:49.281871 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:43:49.281871 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:43:49.281871 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:43:49.281871 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:43:49.281871 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:43:49.281871 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 28 00:43:49.281871 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 28 00:43:49.281871 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 28 00:43:49.281871 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 28 00:43:49.283004 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 28 00:43:49.283004 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 28 00:43:49.283004 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 28 00:43:49.283004 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 28 00:43:49.283004 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:43:49.283004 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:43:49.283004 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:43:49.283004 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:43:49.283004 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:43:49.283004 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:43:49.283004 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:43:49.283004 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:43:49.283004 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:43:49.283004 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:43:49.283004 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:43:49.283004 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:43:49.284285 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:43:49.284285 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:43:49.284285 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:43:49.284285 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:43:49.284285 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 28 00:43:49.284285 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 28 00:43:49.284285 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 28 00:43:49.284285 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 28 00:43:49.284285 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 28 00:43:49.284285 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 28 00:43:49.284285 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 28 00:43:49.284285 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:43:49.284285 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 596/1180] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Aug 28 00:43:49 2022] GET /identity/v3/roles/admin => generated 83 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:43:49.289322 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d4be708-f042-42c5-8c43-6dab2a894040 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:49.392145 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d4be708-f042-42c5-8c43-6dab2a894040 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:49.427437 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e697e3e8-7a9a-4745-bc6a-cefa0d711aae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:49.428068 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e697e3e8-7a9a-4745-bc6a-cefa0d711aae None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:49.428350 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e697e3e8-7a9a-4745-bc6a-cefa0d711aae None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:49.428619 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e697e3e8-7a9a-4745-bc6a-cefa0d711aae None admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:49.429336 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e697e3e8-7a9a-4745-bc6a-cefa0d711aae None admin] RBAC: Adding query filter params (name=admin) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:43:49.429642 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e697e3e8-7a9a-4745-bc6a-cefa0d711aae None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:43:49.431046 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e697e3e8-7a9a-4745-bc6a-cefa0d711aae None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:43:49.440745 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 585/1181] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Aug 28 00:43:49 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:43:49.446940 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a392c29-d911-469e-a906-5c30f38ecd59 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:49.517612 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:43:49.551800 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a392c29-d911-469e-a906-5c30f38ecd59 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:49.585978 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-282c3214-066d-4321-910e-e81de70229b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:49.586589 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-282c3214-066d-4321-910e-e81de70229b4 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:49.586864 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-282c3214-066d-4321-910e-e81de70229b4 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:49.587130 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-282c3214-066d-4321-910e-e81de70229b4 None admin] PATH_INFO: `/v3/users/5853f122a7144d21a01d6867db9cf770` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:49.600998 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-282c3214-066d-4321-910e-e81de70229b4 None admin] RBAC: Authorizing `identity:get_user(user_id=5853f122a7144d21a01d6867db9cf770)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:43:49.602436 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-282c3214-066d-4321-910e-e81de70229b4 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:43:49.604031 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 597/1182] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Aug 28 00:43:49 2022] GET /identity/v3/users/5853f122a7144d21a01d6867db9cf770 => generated 311 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:43:49.610435 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e697e3e8-7a9a-4745-bc6a-cefa0d711aae None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:49.627789 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:43:49.714860 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e697e3e8-7a9a-4745-bc6a-cefa0d711aae None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:49.749959 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b10972c9-cb2b-4137-9d51-b9396d835c56 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:49.750571 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b10972c9-cb2b-4137-9d51-b9396d835c56 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:49.750851 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b10972c9-cb2b-4137-9d51-b9396d835c56 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:49.751118 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b10972c9-cb2b-4137-9d51-b9396d835c56 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:49.820190 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b10972c9-cb2b-4137-9d51-b9396d835c56 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:43:49.821678 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b10972c9-cb2b-4137-9d51-b9396d835c56 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:43:49.836568 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:43:49.927856 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 586/1183] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Aug 28 00:43:49 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 318 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 28 00:43:49.934204 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-282c3214-066d-4321-910e-e81de70229b4 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:50.038212 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-282c3214-066d-4321-910e-e81de70229b4 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:50.063998 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:43:50.073332 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dff6bcad-496b-470f-b40f-b61d16537ac5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:50.074008 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dff6bcad-496b-470f-b40f-b61d16537ac5 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:50.074277 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dff6bcad-496b-470f-b40f-b61d16537ac5 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:50.074542 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dff6bcad-496b-470f-b40f-b61d16537ac5 None admin] PATH_INFO: `/v3/projects/dd5b0eb0b6f840ba9aa189184afbbd11` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:50.075658 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dff6bcad-496b-470f-b40f-b61d16537ac5 None admin] RBAC: Authorizing `identity:get_project(project_id=dd5b0eb0b6f840ba9aa189184afbbd11)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:43:50.077110 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dff6bcad-496b-470f-b40f-b61d16537ac5 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:43:50.079025 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 598/1184] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Aug 28 00:43:49 2022] GET /identity/v3/projects/dd5b0eb0b6f840ba9aa189184afbbd11 => generated 361 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:43:50.085480 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b10972c9-cb2b-4137-9d51-b9396d835c56 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:50.191529 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b10972c9-cb2b-4137-9d51-b9396d835c56 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:50.206306 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:43:50.227471 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56d68c95-2dda-485c-8095-d6ae30a47406 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:50.228069 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56d68c95-2dda-485c-8095-d6ae30a47406 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:50.228339 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56d68c95-2dda-485c-8095-d6ae30a47406 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:50.228601 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56d68c95-2dda-485c-8095-d6ae30a47406 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:50.229359 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56d68c95-2dda-485c-8095-d6ae30a47406 None admin] RBAC: Adding query filter params (role.id=f08d6043261747e3b4763c744eef7dfb, scope.project.id=dd5b0eb0b6f840ba9aa189184afbbd11, user.id=5853f122a7144d21a01d6867db9cf770) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:43:50.229658 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56d68c95-2dda-485c-8095-d6ae30a47406 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:43:50.231122 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56d68c95-2dda-485c-8095-d6ae30a47406 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:43:50.236584 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 587/1185] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Aug 28 00:43:50 2022] GET /identity/v3/role_assignments?user.id=5853f122a7144d21a01d6867db9cf770&scope.project.id=dd5b0eb0b6f840ba9aa189184afbbd11&role.id=f08d6043261747e3b4763c744eef7dfb => generated 259 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:43:52.107237 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0f8e925-efe5-4bbc-8c34-97a0ab507499 None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:52.107796 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0f8e925-efe5-4bbc-8c34-97a0ab507499 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:52.107796 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0f8e925-efe5-4bbc-8c34-97a0ab507499 None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:52.109406 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 599/1186] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:43:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:43:52.114647 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9647c0a9-2387-449f-9b97-f27631d70152 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:52.115141 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9647c0a9-2387-449f-9b97-f27631d70152 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:52.115141 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9647c0a9-2387-449f-9b97-f27631d70152 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:52.134172 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-9647c0a9-2387-449f-9b97-f27631d70152 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:43:52.135367 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-9647c0a9-2387-449f-9b97-f27631d70152 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:43:52.203205 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 588/1187] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:43:52 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 28 00:43:52.466601 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0f8e925-efe5-4bbc-8c34-97a0ab507499 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:52.575506 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0f8e925-efe5-4bbc-8c34-97a0ab507499 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:52.612019 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65157e0a-4df9-43d4-b310-70857ffc2907 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:52.612638 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65157e0a-4df9-43d4-b310-70857ffc2907 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:52.612912 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65157e0a-4df9-43d4-b310-70857ffc2907 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:52.613210 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65157e0a-4df9-43d4-b310-70857ffc2907 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:52.623927 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65157e0a-4df9-43d4-b310-70857ffc2907 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 28 00:43:52.628827 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-65157e0a-4df9-43d4-b310-70857ffc2907 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 28 00:43:52.628827 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:43:52.628827 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:43:52.628827 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:43:52.628827 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:43:52.628827 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:43:52.628827 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:43:52.628827 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:43:52.628827 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:43:52.628827 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:43:52.628827 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:43:52.628827 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:43:52.628827 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 28 00:43:52.628827 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 28 00:43:52.628827 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 28 00:43:52.628827 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 28 00:43:52.629883 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 28 00:43:52.629883 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 28 00:43:52.629883 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 28 00:43:52.629883 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 28 00:43:52.629883 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:43:52.629883 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:43:52.629883 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:43:52.629883 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:43:52.629883 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:43:52.629883 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:43:52.629883 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:43:52.629883 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:43:52.629883 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:43:52.629883 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:43:52.629883 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:43:52.629883 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:43:52.630903 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:43:52.630903 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:43:52.630903 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:43:52.630903 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:43:52.630903 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 28 00:43:52.630903 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 28 00:43:52.630903 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 28 00:43:52.630903 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 28 00:43:52.630903 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 28 00:43:52.630903 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 28 00:43:52.630903 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 28 00:43:52.630903 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:43:52.630903 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 600/1188] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Aug 28 00:43:52 2022] GET /identity/v3/roles/admin => generated 83 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:43:52.636321 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9647c0a9-2387-449f-9b97-f27631d70152 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:52.740083 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9647c0a9-2387-449f-9b97-f27631d70152 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:52.750052 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:43:52.777593 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b263fffa-beb0-47e2-8bb2-4478510789cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:52.778230 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b263fffa-beb0-47e2-8bb2-4478510789cc None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:52.778483 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b263fffa-beb0-47e2-8bb2-4478510789cc None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:52.778756 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b263fffa-beb0-47e2-8bb2-4478510789cc None admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:52.779414 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b263fffa-beb0-47e2-8bb2-4478510789cc None admin] RBAC: Adding query filter params (name=admin) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:43:52.779717 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b263fffa-beb0-47e2-8bb2-4478510789cc None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:43:52.781113 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b263fffa-beb0-47e2-8bb2-4478510789cc None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:43:52.789439 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 589/1189] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Aug 28 00:43:52 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 28 00:43:52.796122 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65157e0a-4df9-43d4-b310-70857ffc2907 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:52.853963 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:43:52.899270 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65157e0a-4df9-43d4-b310-70857ffc2907 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:52.935284 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-125b32b3-78f5-49a0-ade3-7870788fef07 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:52.935907 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-125b32b3-78f5-49a0-ade3-7870788fef07 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:52.936178 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-125b32b3-78f5-49a0-ade3-7870788fef07 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:52.936439 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-125b32b3-78f5-49a0-ade3-7870788fef07 None admin] PATH_INFO: `/v3/users/5853f122a7144d21a01d6867db9cf770` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:52.937606 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-125b32b3-78f5-49a0-ade3-7870788fef07 None admin] RBAC: Authorizing `identity:get_user(user_id=5853f122a7144d21a01d6867db9cf770)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:43:52.939035 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-125b32b3-78f5-49a0-ade3-7870788fef07 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:43:52.940594 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 601/1190] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Aug 28 00:43:52 2022] GET /identity/v3/users/5853f122a7144d21a01d6867db9cf770 => generated 311 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:43:52.946647 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b263fffa-beb0-47e2-8bb2-4478510789cc None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:53.052065 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b263fffa-beb0-47e2-8bb2-4478510789cc None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:53.087476 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85e6ea89-6521-4d7e-955a-1654ce04ded9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:53.088103 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85e6ea89-6521-4d7e-955a-1654ce04ded9 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:53.088381 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85e6ea89-6521-4d7e-955a-1654ce04ded9 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:53.088655 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85e6ea89-6521-4d7e-955a-1654ce04ded9 None admin] PATH_INFO: `/v3/projects/dd5b0eb0b6f840ba9aa189184afbbd11` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:53.089862 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85e6ea89-6521-4d7e-955a-1654ce04ded9 None admin] RBAC: Authorizing `identity:get_project(project_id=dd5b0eb0b6f840ba9aa189184afbbd11)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:43:53.091352 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85e6ea89-6521-4d7e-955a-1654ce04ded9 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:43:53.093014 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 590/1191] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Aug 28 00:43:52 2022] GET /identity/v3/projects/dd5b0eb0b6f840ba9aa189184afbbd11 => generated 361 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:43:53.099300 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-125b32b3-78f5-49a0-ade3-7870788fef07 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:53.205385 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-125b32b3-78f5-49a0-ade3-7870788fef07 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:53.228505 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:43:53.242456 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f166e72-fdd3-467d-af28-967f6b82d628 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:53.243131 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f166e72-fdd3-467d-af28-967f6b82d628 None admin] REQUEST_METHOD: `PUT` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:53.243400 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f166e72-fdd3-467d-af28-967f6b82d628 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:53.243682 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f166e72-fdd3-467d-af28-967f6b82d628 None admin] PATH_INFO: `/v3/projects/dd5b0eb0b6f840ba9aa189184afbbd11/users/5853f122a7144d21a01d6867db9cf770/roles/f08d6043261747e3b4763c744eef7dfb` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:53.245805 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f166e72-fdd3-467d-af28-967f6b82d628 None admin] RBAC: Authorizing `identity:create_grant(project_id=dd5b0eb0b6f840ba9aa189184afbbd11, user_id=5853f122a7144d21a01d6867db9cf770, role_id=f08d6043261747e3b4763c744eef7dfb)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:43:53.247267 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f166e72-fdd3-467d-af28-967f6b82d628 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:43:53.262183 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 602/1192] 10.222.0.42 () {58 vars in 1439 bytes} [Sun Aug 28 00:43:53 2022] PUT /identity/v3/projects/dd5b0eb0b6f840ba9aa189184afbbd11/users/5853f122a7144d21a01d6867db9cf770/roles/f08d6043261747e3b4763c744eef7dfb => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:43:55.123078 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb55fbef-0c95-4eb6-95fc-5b251f4e2fc2 None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:55.123523 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb55fbef-0c95-4eb6-95fc-5b251f4e2fc2 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:55.123614 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb55fbef-0c95-4eb6-95fc-5b251f4e2fc2 None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:55.125865 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 591/1193] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:43:55 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:43:55.131609 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c198ecaa-4273-4045-9767-de4cfe6631ea None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:55.132138 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c198ecaa-4273-4045-9767-de4cfe6631ea None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:55.132138 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c198ecaa-4273-4045-9767-de4cfe6631ea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:55.151522 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-c198ecaa-4273-4045-9767-de4cfe6631ea None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:43:55.152885 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-c198ecaa-4273-4045-9767-de4cfe6631ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:43:55.221613 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 603/1194] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:43:55 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 28 00:43:55.482845 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb55fbef-0c95-4eb6-95fc-5b251f4e2fc2 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:55.525570 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:43:55.589299 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb55fbef-0c95-4eb6-95fc-5b251f4e2fc2 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:55.624979 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57adb6d8-ffaf-4451-93c5-ed0aa0148cbf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:55.625607 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57adb6d8-ffaf-4451-93c5-ed0aa0148cbf None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:55.625889 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57adb6d8-ffaf-4451-93c5-ed0aa0148cbf None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:55.626143 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57adb6d8-ffaf-4451-93c5-ed0aa0148cbf None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:55.636831 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57adb6d8-ffaf-4451-93c5-ed0aa0148cbf None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 28 00:43:55.641789 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-57adb6d8-ffaf-4451-93c5-ed0aa0148cbf None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 28 00:43:55.641789 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:43:55.641789 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:43:55.641789 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:43:55.641789 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:43:55.641789 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:43:55.641789 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:43:55.641789 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:43:55.641789 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:43:55.641789 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:43:55.641789 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:43:55.641789 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:43:55.641789 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 28 00:43:55.641789 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 28 00:43:55.641789 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 28 00:43:55.641789 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 28 00:43:55.642903 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 28 00:43:55.642903 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 28 00:43:55.642903 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 28 00:43:55.642903 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 28 00:43:55.642903 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:43:55.642903 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:43:55.642903 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:43:55.642903 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:43:55.642903 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:43:55.642903 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:43:55.642903 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:43:55.642903 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:43:55.642903 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:43:55.642903 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:43:55.642903 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:43:55.642903 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:43:55.643755 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:43:55.643755 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:43:55.643755 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:43:55.643755 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:43:55.643755 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 28 00:43:55.643755 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 28 00:43:55.643755 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 28 00:43:55.643755 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 28 00:43:55.643755 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 28 00:43:55.643755 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 28 00:43:55.643755 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 28 00:43:55.643755 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:43:55.643755 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 592/1195] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Aug 28 00:43:55 2022] GET /identity/v3/roles/admin => generated 83 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:43:55.649235 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c198ecaa-4273-4045-9767-de4cfe6631ea None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:55.753136 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c198ecaa-4273-4045-9767-de4cfe6631ea None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:55.766807 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:43:55.790621 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2146d3ba-6f84-4505-8427-c6838755e2c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:55.791233 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2146d3ba-6f84-4505-8427-c6838755e2c1 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:55.791512 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2146d3ba-6f84-4505-8427-c6838755e2c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:55.791776 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2146d3ba-6f84-4505-8427-c6838755e2c1 None admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:55.792504 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2146d3ba-6f84-4505-8427-c6838755e2c1 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:43:55.792805 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2146d3ba-6f84-4505-8427-c6838755e2c1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:43:55.794211 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2146d3ba-6f84-4505-8427-c6838755e2c1 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:43:55.802419 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 604/1196] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Aug 28 00:43:55 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:43:55.808540 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57adb6d8-ffaf-4451-93c5-ed0aa0148cbf None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:55.906368 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:43:55.910859 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57adb6d8-ffaf-4451-93c5-ed0aa0148cbf None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:55.948601 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f1449d9-b52d-402c-8036-af18a6952021 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:55.949719 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f1449d9-b52d-402c-8036-af18a6952021 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:55.950347 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f1449d9-b52d-402c-8036-af18a6952021 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:55.950660 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f1449d9-b52d-402c-8036-af18a6952021 None admin] PATH_INFO: `/v3/users/5853f122a7144d21a01d6867db9cf770` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:55.951941 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f1449d9-b52d-402c-8036-af18a6952021 None admin] RBAC: Authorizing `identity:get_user(user_id=5853f122a7144d21a01d6867db9cf770)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:43:55.953438 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f1449d9-b52d-402c-8036-af18a6952021 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:43:55.955360 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 593/1197] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Aug 28 00:43:55 2022] GET /identity/v3/users/5853f122a7144d21a01d6867db9cf770 => generated 311 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:43:55.961662 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2146d3ba-6f84-4505-8427-c6838755e2c1 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:56.067430 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2146d3ba-6f84-4505-8427-c6838755e2c1 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:56.101903 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b044a28-5912-4060-9928-95cefd984153 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:56.102502 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b044a28-5912-4060-9928-95cefd984153 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:56.102776 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b044a28-5912-4060-9928-95cefd984153 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:56.103038 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b044a28-5912-4060-9928-95cefd984153 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:56.147569 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:43:56.174923 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b044a28-5912-4060-9928-95cefd984153 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:43:56.176381 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b044a28-5912-4060-9928-95cefd984153 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:43:56.277219 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 605/1198] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Aug 28 00:43:55 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 316 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 28 00:43:56.283394 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f1449d9-b52d-402c-8036-af18a6952021 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:56.394676 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f1449d9-b52d-402c-8036-af18a6952021 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:56.429955 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4093f5a8-6f9f-4125-a910-e32d02b437a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:56.430560 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4093f5a8-6f9f-4125-a910-e32d02b437a8 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:56.430840 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4093f5a8-6f9f-4125-a910-e32d02b437a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:56.431137 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4093f5a8-6f9f-4125-a910-e32d02b437a8 None admin] PATH_INFO: `/v3/projects/dd5b0eb0b6f840ba9aa189184afbbd11` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:56.432282 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4093f5a8-6f9f-4125-a910-e32d02b437a8 None admin] RBAC: Authorizing `identity:get_project(project_id=dd5b0eb0b6f840ba9aa189184afbbd11)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:43:56.433772 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4093f5a8-6f9f-4125-a910-e32d02b437a8 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:43:56.435418 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 594/1199] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Aug 28 00:43:56 2022] GET /identity/v3/projects/dd5b0eb0b6f840ba9aa189184afbbd11 => generated 361 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:43:56.442108 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b044a28-5912-4060-9928-95cefd984153 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:43:56.526012 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:43:56.555005 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b044a28-5912-4060-9928-95cefd984153 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:43:56.591387 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05a9d323-9156-4865-82f8-54a8623d5888 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:43:56.591991 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05a9d323-9156-4865-82f8-54a8623d5888 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:43:56.592265 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05a9d323-9156-4865-82f8-54a8623d5888 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:43:56.592533 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05a9d323-9156-4865-82f8-54a8623d5888 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:43:56.593291 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05a9d323-9156-4865-82f8-54a8623d5888 None admin] RBAC: Adding query filter params (role.id=f08d6043261747e3b4763c744eef7dfb, scope.project.id=dd5b0eb0b6f840ba9aa189184afbbd11, user.id=5853f122a7144d21a01d6867db9cf770) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:43:56.593594 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05a9d323-9156-4865-82f8-54a8623d5888 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:43:56.595036 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05a9d323-9156-4865-82f8-54a8623d5888 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:43:56.600981 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 606/1200] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Aug 28 00:43:56 2022] GET /identity/v3/role_assignments?user.id=5853f122a7144d21a01d6867db9cf770&scope.project.id=dd5b0eb0b6f840ba9aa189184afbbd11&role.id=f08d6043261747e3b4763c744eef7dfb => generated 608 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:44:06.425197 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=83 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:44:06.598704 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=31 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:44:55.353744 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5342e693-f264-4617-b2d8-94de8d656895 None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:44:55.353744 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5342e693-f264-4617-b2d8-94de8d656895 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:44:55.353744 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5342e693-f264-4617-b2d8-94de8d656895 None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:44:55.353744 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 595/1201] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:44:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 28 00:44:55.361525 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-addf3753-ae2a-4cc5-adde-babf99519069 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:44:55.362409 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-addf3753-ae2a-4cc5-adde-babf99519069 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:44:55.362409 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-addf3753-ae2a-4cc5-adde-babf99519069 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:44:55.388818 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-addf3753-ae2a-4cc5-adde-babf99519069 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:44:55.390255 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-addf3753-ae2a-4cc5-adde-babf99519069 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:44:55.523239 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 607/1202] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:44:55 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 28 00:44:55.847453 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5342e693-f264-4617-b2d8-94de8d656895 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:44:56.035616 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5342e693-f264-4617-b2d8-94de8d656895 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:44:56.095181 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ae0db25-ecee-42c1-9e1e-bd13e5c50a32 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:44:56.096049 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ae0db25-ecee-42c1-9e1e-bd13e5c50a32 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:44:56.096463 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ae0db25-ecee-42c1-9e1e-bd13e5c50a32 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:44:56.096878 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ae0db25-ecee-42c1-9e1e-bd13e5c50a32 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:44:56.118410 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ae0db25-ecee-42c1-9e1e-bd13e5c50a32 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:44:56.119847 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ae0db25-ecee-42c1-9e1e-bd13e5c50a32 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:44:56.146502 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-0ae0db25-ecee-42c1-9e1e-bd13e5c50a32 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:44:56.146502 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:44:56.146502 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:44:56.146502 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:44:56.146502 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:44:56.146502 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:44:56.146502 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:44:56.146502 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:44:56.146502 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:44:56.146502 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:44:56.146502 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:44:56.146502 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:44:56.146502 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 28 00:44:56.146502 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 28 00:44:56.146502 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 28 00:44:56.146502 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 28 00:44:56.148799 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:44:56.148799 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:44:56.148799 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:44:56.148799 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:44:56.148799 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:44:56.148799 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:44:56.148799 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:44:56.148799 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:44:56.148799 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:44:56.148799 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:44:56.148799 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:44:56.148799 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:44:56.148799 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:44:56.148799 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:44:56.148799 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:44:56.148799 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:44:56.148799 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 28 00:44:56.150813 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 28 00:44:56.150813 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:44:56.150813 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:44:56.150813 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 596/1203] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:44:55 2022] GET /identity/v3/domains/Default => generated 87 bytes in 302 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:44:56.157336 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-addf3753-ae2a-4cc5-adde-babf99519069 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:44:56.356929 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-addf3753-ae2a-4cc5-adde-babf99519069 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:44:56.418880 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:44:56.429616 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-336a07a9-a124-459d-a9c4-daf70232d2ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:44:56.430782 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-336a07a9-a124-459d-a9c4-daf70232d2ea None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:44:56.431282 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-336a07a9-a124-459d-a9c4-daf70232d2ea None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:44:56.431695 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-336a07a9-a124-459d-a9c4-daf70232d2ea None admin] PATH_INFO: `/v3/domains` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:44:56.432813 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-336a07a9-a124-459d-a9c4-daf70232d2ea None admin] RBAC: Adding query filter params (name=Default) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:44:56.433305 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-336a07a9-a124-459d-a9c4-daf70232d2ea None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:44:56.435562 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-336a07a9-a124-459d-a9c4-daf70232d2ea None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:44:56.456835 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 608/1204] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Aug 28 00:44:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 301 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:44:56.466896 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ae0db25-ecee-42c1-9e1e-bd13e5c50a32 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:44:56.563696 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:44:56.635629 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ae0db25-ecee-42c1-9e1e-bd13e5c50a32 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:44:56.685665 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56a64af0-0d74-4050-b130-ab17de067d53 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:44:56.686684 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a64af0-0d74-4050-b130-ab17de067d53 None admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:44:56.687158 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a64af0-0d74-4050-b130-ab17de067d53 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:44:56.687535 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a64af0-0d74-4050-b130-ab17de067d53 None admin] PATH_INFO: `/v3/users` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:44:56.688608 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56a64af0-0d74-4050-b130-ab17de067d53 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:44:56.690741 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56a64af0-0d74-4050-b130-ab17de067d53 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:44:56.734242 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 597/1205] 10.222.0.42 () {60 vars in 1132 bytes} [Sun Aug 28 00:44:56 2022] POST /identity/v3/users => generated 251 bytes in 268 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 28 00:44:59.024829 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fa6b673-c78c-4bd2-bf42-4a5e349eabcc None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:44:59.025470 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fa6b673-c78c-4bd2-bf42-4a5e349eabcc None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:44:59.025778 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fa6b673-c78c-4bd2-bf42-4a5e349eabcc None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:44:59.027997 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 609/1206] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:44:59 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:44:59.037434 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a310f635-b1d6-4f44-9bde-153f0e6fba9b None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:44:59.038001 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a310f635-b1d6-4f44-9bde-153f0e6fba9b None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:44:59.038600 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a310f635-b1d6-4f44-9bde-153f0e6fba9b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:44:59.077092 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-a310f635-b1d6-4f44-9bde-153f0e6fba9b None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:44:59.078795 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-a310f635-b1d6-4f44-9bde-153f0e6fba9b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:44:59.211302 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 598/1207] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:44:59 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 28 00:44:59.697439 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fa6b673-c78c-4bd2-bf42-4a5e349eabcc None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:44:59.892293 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fa6b673-c78c-4bd2-bf42-4a5e349eabcc None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:44:59.953635 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dbdd6e8-f7d7-4325-847c-6c91bb1a11e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:44:59.953635 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dbdd6e8-f7d7-4325-847c-6c91bb1a11e6 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:44:59.953635 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dbdd6e8-f7d7-4325-847c-6c91bb1a11e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:44:59.954263 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dbdd6e8-f7d7-4325-847c-6c91bb1a11e6 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:44:59.975052 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dbdd6e8-f7d7-4325-847c-6c91bb1a11e6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 28 00:44:59.984650 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-0dbdd6e8-f7d7-4325-847c-6c91bb1a11e6 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 28 00:44:59.984650 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:44:59.984650 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:44:59.984650 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:44:59.984650 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:44:59.984650 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:44:59.984650 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:44:59.984650 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:44:59.984650 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:44:59.984650 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:44:59.984650 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:44:59.984650 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:44:59.984650 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 28 00:44:59.984650 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 28 00:44:59.984650 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 28 00:44:59.984650 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 28 00:44:59.986831 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 28 00:44:59.986831 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 28 00:44:59.986831 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 28 00:44:59.986831 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 28 00:44:59.986831 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:44:59.986831 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:44:59.986831 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:44:59.986831 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:44:59.986831 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:44:59.986831 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:44:59.986831 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:44:59.986831 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:44:59.986831 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:44:59.986831 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:44:59.986831 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:44:59.986831 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:44:59.988439 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:44:59.988439 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:44:59.988439 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:44:59.988439 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:44:59.988439 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 28 00:44:59.988439 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 28 00:44:59.988439 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 28 00:44:59.988439 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 28 00:44:59.988439 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 28 00:44:59.988439 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 28 00:44:59.988439 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 28 00:44:59.988439 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:44:59.988439 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 610/1208] 10.222.0.42 () {56 vars in 1102 bytes} [Sun Aug 28 00:44:59 2022] GET /identity/v3/roles/service => generated 85 bytes in 290 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:44:59.996861 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a310f635-b1d6-4f44-9bde-153f0e6fba9b None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:45:00.059097 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:45:00.180490 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a310f635-b1d6-4f44-9bde-153f0e6fba9b None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:45:00.241935 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d82298a-1bb1-472e-8e9f-8db88095fe0a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:45:00.242873 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d82298a-1bb1-472e-8e9f-8db88095fe0a None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:45:00.243430 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d82298a-1bb1-472e-8e9f-8db88095fe0a None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:45:00.243888 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d82298a-1bb1-472e-8e9f-8db88095fe0a None admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:45:00.244761 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d82298a-1bb1-472e-8e9f-8db88095fe0a None admin] RBAC: Adding query filter params (name=service) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:45:00.245381 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d82298a-1bb1-472e-8e9f-8db88095fe0a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:45:00.247798 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d82298a-1bb1-472e-8e9f-8db88095fe0a None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:45:00.263643 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 599/1209] 10.222.0.42 () {56 vars in 1103 bytes} [Sun Aug 28 00:44:59 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 268 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:45:00.270447 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dbdd6e8-f7d7-4325-847c-6c91bb1a11e6 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:45:00.367355 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:45:00.464604 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dbdd6e8-f7d7-4325-847c-6c91bb1a11e6 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:45:00.556090 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0240842-37d3-4af2-8b16-017148f02bad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:45:00.557445 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0240842-37d3-4af2-8b16-017148f02bad None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:45:00.558696 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0240842-37d3-4af2-8b16-017148f02bad None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:45:00.558696 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0240842-37d3-4af2-8b16-017148f02bad None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:45:00.578299 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0240842-37d3-4af2-8b16-017148f02bad None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:45:00.580192 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0240842-37d3-4af2-8b16-017148f02bad None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:45:00.603242 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-b0240842-37d3-4af2-8b16-017148f02bad None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:45:00.603242 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:45:00.603242 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:45:00.603242 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:45:00.603242 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:45:00.603242 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:45:00.603242 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:45:00.603242 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:45:00.603242 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:45:00.603242 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:45:00.603242 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:45:00.603242 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:45:00.603242 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 28 00:45:00.603242 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 28 00:45:00.603242 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 28 00:45:00.603242 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 28 00:45:00.605152 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:45:00.605152 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:45:00.605152 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:45:00.605152 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:45:00.605152 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:45:00.605152 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:45:00.605152 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:45:00.605152 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:45:00.605152 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:45:00.605152 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:45:00.605152 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:45:00.605152 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:45:00.605152 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:45:00.605152 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:45:00.605152 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:45:00.605152 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:45:00.605152 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 28 00:45:00.606897 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 28 00:45:00.606897 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:45:00.606897 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:45:00.606897 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 611/1210] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:45:00 2022] GET /identity/v3/domains/Default => generated 87 bytes in 335 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 28 00:45:00.612022 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d82298a-1bb1-472e-8e9f-8db88095fe0a None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:45:00.839555 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:45:00.845918 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d82298a-1bb1-472e-8e9f-8db88095fe0a None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:45:00.936072 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c954b726-7399-4be1-984d-856e6f31c0f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:45:00.937360 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c954b726-7399-4be1-984d-856e6f31c0f1 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:45:00.937940 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c954b726-7399-4be1-984d-856e6f31c0f1 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:45:00.938465 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c954b726-7399-4be1-984d-856e6f31c0f1 None admin] PATH_INFO: `/v3/domains` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:45:00.939732 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c954b726-7399-4be1-984d-856e6f31c0f1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:45:00.940348 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c954b726-7399-4be1-984d-856e6f31c0f1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:45:00.943196 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c954b726-7399-4be1-984d-856e6f31c0f1 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:45:00.960156 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 600/1211] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Aug 28 00:45:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 349 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 28 00:45:00.970492 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0240842-37d3-4af2-8b16-017148f02bad None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:45:01.167894 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0240842-37d3-4af2-8b16-017148f02bad None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:45:01.173778 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:45:01.249414 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c2d81ab-a1b5-4ec0-adbf-e8396df34a1e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:45:01.250585 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c2d81ab-a1b5-4ec0-adbf-e8396df34a1e None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:45:01.251382 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c2d81ab-a1b5-4ec0-adbf-e8396df34a1e None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:45:01.251382 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c2d81ab-a1b5-4ec0-adbf-e8396df34a1e None admin] PATH_INFO: `/v3/users/placement` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:45:01.261790 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c2d81ab-a1b5-4ec0-adbf-e8396df34a1e None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:45:01.263939 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c2d81ab-a1b5-4ec0-adbf-e8396df34a1e None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:45:01.279313 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-3c2d81ab-a1b5-4ec0-adbf-e8396df34a1e None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Aug 28 00:45:01.279313 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:45:01.279313 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:45:01.279313 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:45:01.279313 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:45:01.279313 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:45:01.279313 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:45:01.279313 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:45:01.279313 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:45:01.279313 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:45:01.279313 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:45:01.279313 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:45:01.279313 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 28 00:45:01.279313 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 28 00:45:01.279313 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 28 00:45:01.279313 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 28 00:45:01.282302 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:45:01.282302 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:45:01.282302 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 28 00:45:01.282302 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 28 00:45:01.282302 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 28 00:45:01.282302 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 28 00:45:01.282302 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:45:01.282302 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:45:01.282302 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:45:01.282302 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:45:01.282302 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:45:01.282302 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:45:01.282302 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:45:01.282302 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:45:01.282302 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:45:01.282302 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:45:01.282302 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:45:01.282302 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:45:01.284117 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:45:01.284117 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:45:01.284117 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 28 00:45:01.284117 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 28 00:45:01.284117 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 28 00:45:01.284117 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 28 00:45:01.284117 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 28 00:45:01.284117 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 28 00:45:01.284117 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Aug 28 00:45:01.284117 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:45:01.284117 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 612/1212] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:45:00 2022] GET /identity/v3/users/placement => generated 87 bytes in 312 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 28 00:45:01.287813 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c954b726-7399-4be1-984d-856e6f31c0f1 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:45:01.461427 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c954b726-7399-4be1-984d-856e6f31c0f1 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:45:01.539069 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3343d547-0b6c-4b8a-b2f4-2cd8b01343b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:45:01.540627 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3343d547-0b6c-4b8a-b2f4-2cd8b01343b8 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:45:01.541135 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3343d547-0b6c-4b8a-b2f4-2cd8b01343b8 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:45:01.541766 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3343d547-0b6c-4b8a-b2f4-2cd8b01343b8 None admin] PATH_INFO: `/v3/users` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:45:01.543230 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3343d547-0b6c-4b8a-b2f4-2cd8b01343b8 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:45:01.543769 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3343d547-0b6c-4b8a-b2f4-2cd8b01343b8 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:45:01.546521 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3343d547-0b6c-4b8a-b2f4-2cd8b01343b8 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:45:01.588138 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 601/1213] 10.222.0.42 () {56 vars in 1143 bytes} [Sun Aug 28 00:45:01 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 297 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:45:01.590469 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c2d81ab-a1b5-4ec0-adbf-e8396df34a1e None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:45:01.795907 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c2d81ab-a1b5-4ec0-adbf-e8396df34a1e None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:45:01.836090 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af59329c-4003-4160-a893-d43b04c8c9c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:45:01.836751 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af59329c-4003-4160-a893-d43b04c8c9c2 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:45:01.837049 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af59329c-4003-4160-a893-d43b04c8c9c2 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:45:01.837317 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af59329c-4003-4160-a893-d43b04c8c9c2 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:45:01.871657 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:45:01.915442 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af59329c-4003-4160-a893-d43b04c8c9c2 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:45:01.917076 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af59329c-4003-4160-a893-d43b04c8c9c2 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:45:02.061173 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 613/1214] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Aug 28 00:45:01 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 472 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 28 00:45:02.068302 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3343d547-0b6c-4b8a-b2f4-2cd8b01343b8 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:45:02.092832 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:45:02.190494 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3343d547-0b6c-4b8a-b2f4-2cd8b01343b8 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:45:02.234214 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92eb00fc-6448-4774-aa6c-4aa42f39c379 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:45:02.235134 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92eb00fc-6448-4774-aa6c-4aa42f39c379 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:45:02.235642 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92eb00fc-6448-4774-aa6c-4aa42f39c379 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:45:02.235946 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92eb00fc-6448-4774-aa6c-4aa42f39c379 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:45:02.250478 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92eb00fc-6448-4774-aa6c-4aa42f39c379 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:45:02.252047 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92eb00fc-6448-4774-aa6c-4aa42f39c379 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:45:02.270333 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-92eb00fc-6448-4774-aa6c-4aa42f39c379 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:45:02.270333 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:45:02.270333 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:45:02.270333 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:45:02.270333 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:45:02.270333 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:45:02.270333 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:45:02.270333 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:45:02.270333 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:45:02.270333 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:45:02.270333 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:45:02.270333 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:45:02.270333 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 28 00:45:02.270333 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 28 00:45:02.270333 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 28 00:45:02.270333 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 28 00:45:02.272549 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:45:02.272549 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:45:02.272549 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:45:02.272549 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:45:02.272549 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:45:02.272549 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:45:02.272549 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:45:02.272549 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:45:02.272549 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:45:02.272549 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:45:02.272549 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:45:02.272549 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:45:02.272549 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:45:02.272549 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:45:02.272549 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:45:02.272549 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:45:02.272549 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 28 00:45:02.274385 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 28 00:45:02.274385 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:45:02.274385 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:45:02.277881 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af59329c-4003-4160-a893-d43b04c8c9c2 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:45:02.280138 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 602/1215] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:45:02 2022] GET /identity/v3/domains/Default => generated 87 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 28 00:45:02.352596 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:45:02.414244 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af59329c-4003-4160-a893-d43b04c8c9c2 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:45:02.466867 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72a1fae5-d912-4a04-8a2f-06fa2ec487e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:45:02.467533 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a1fae5-d912-4a04-8a2f-06fa2ec487e3 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:45:02.467906 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a1fae5-d912-4a04-8a2f-06fa2ec487e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:45:02.468326 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a1fae5-d912-4a04-8a2f-06fa2ec487e3 None admin] PATH_INFO: `/v3/domains` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:45:02.469186 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72a1fae5-d912-4a04-8a2f-06fa2ec487e3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:45:02.469686 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72a1fae5-d912-4a04-8a2f-06fa2ec487e3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:45:02.471871 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72a1fae5-d912-4a04-8a2f-06fa2ec487e3 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:45:02.487964 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 614/1216] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Aug 28 00:45:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:45:02.494142 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92eb00fc-6448-4774-aa6c-4aa42f39c379 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:45:02.592689 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:45:02.634673 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92eb00fc-6448-4774-aa6c-4aa42f39c379 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:45:02.686086 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b66698c-c569-46a8-9fab-935a369fd4ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:45:02.686086 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b66698c-c569-46a8-9fab-935a369fd4ff None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:45:02.686086 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b66698c-c569-46a8-9fab-935a369fd4ff None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:45:02.686552 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b66698c-c569-46a8-9fab-935a369fd4ff None admin] PATH_INFO: `/v3/projects/service` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:45:02.696060 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b66698c-c569-46a8-9fab-935a369fd4ff None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:45:02.697820 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b66698c-c569-46a8-9fab-935a369fd4ff None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:45:02.711140 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-5b66698c-c569-46a8-9fab-935a369fd4ff None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 28 00:45:02.711140 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:45:02.711140 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:45:02.711140 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:45:02.711140 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:45:02.711140 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:45:02.711140 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:45:02.711140 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:45:02.711140 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:45:02.711140 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:45:02.711140 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:45:02.711140 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:45:02.711140 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 28 00:45:02.711140 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 28 00:45:02.711140 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 28 00:45:02.713208 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 28 00:45:02.713208 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:45:02.713208 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:45:02.713208 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:45:02.713208 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:45:02.713208 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:45:02.713208 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:45:02.713208 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:45:02.713208 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:45:02.713208 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:45:02.713208 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:45:02.713208 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:45:02.713208 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:45:02.713208 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:45:02.713208 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:45:02.713208 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:45:02.713208 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:45:02.715163 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 28 00:45:02.715163 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 28 00:45:02.715163 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 28 00:45:02.715163 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 28 00:45:02.715163 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 28 00:45:02.715163 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 28 00:45:02.715163 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 28 00:45:02.715163 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:45:02.715163 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 603/1217] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Aug 28 00:45:02 2022] GET /identity/v3/projects/service => generated 88 bytes in 219 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 28 00:45:02.724077 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72a1fae5-d912-4a04-8a2f-06fa2ec487e3 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:45:02.850070 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72a1fae5-d912-4a04-8a2f-06fa2ec487e3 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:45:02.882984 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:45:02.899042 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3ced804-3aca-48d9-a823-1d116e78715b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:45:02.899989 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3ced804-3aca-48d9-a823-1d116e78715b None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:45:02.900385 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3ced804-3aca-48d9-a823-1d116e78715b None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:45:02.900648 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3ced804-3aca-48d9-a823-1d116e78715b None admin] PATH_INFO: `/v3/projects` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:45:02.901768 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3ced804-3aca-48d9-a823-1d116e78715b None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:45:02.902011 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3ced804-3aca-48d9-a823-1d116e78715b None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:45:02.903884 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3ced804-3aca-48d9-a823-1d116e78715b None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:45:02.919782 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 615/1218] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Aug 28 00:45:02 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:45:02.927245 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b66698c-c569-46a8-9fab-935a369fd4ff None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:45:03.080035 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b66698c-c569-46a8-9fab-935a369fd4ff None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:45:03.123334 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:45:03.135069 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5272a619-2484-463d-8c77-f4875cc3f5e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:45:03.135847 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5272a619-2484-463d-8c77-f4875cc3f5e9 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:45:03.136232 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5272a619-2484-463d-8c77-f4875cc3f5e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:45:03.136648 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5272a619-2484-463d-8c77-f4875cc3f5e9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:45:03.137655 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5272a619-2484-463d-8c77-f4875cc3f5e9 None admin] RBAC: Adding query filter params (role.id=348070fea3dd4cc2a6e00e29fe82cc25, scope.project.id=f53e6fdb369b480fbebb4086a505df67, user.id=c2a3834fa6894dada3291e4781bbd553) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:45:03.138176 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5272a619-2484-463d-8c77-f4875cc3f5e9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:45:03.140798 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5272a619-2484-463d-8c77-f4875cc3f5e9 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:45:03.151917 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 604/1219] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Aug 28 00:45:02 2022] GET /identity/v3/role_assignments?user.id=c2a3834fa6894dada3291e4781bbd553&scope.project.id=f53e6fdb369b480fbebb4086a505df67&role.id=348070fea3dd4cc2a6e00e29fe82cc25 => generated 259 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:45:05.204565 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4a445b6-dfa8-4395-8638-e1703697ab7a None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:45:05.205230 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4a445b6-dfa8-4395-8638-e1703697ab7a None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:45:05.205401 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4a445b6-dfa8-4395-8638-e1703697ab7a None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:45:05.206865 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 616/1220] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:45:05 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:45:05.214188 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff86d36-1718-4f4c-af1f-3ead392d7bc0 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:45:05.214877 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff86d36-1718-4f4c-af1f-3ead392d7bc0 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:45:05.214877 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff86d36-1718-4f4c-af1f-3ead392d7bc0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:45:05.237571 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-bff86d36-1718-4f4c-af1f-3ead392d7bc0 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:45:05.239778 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-bff86d36-1718-4f4c-af1f-3ead392d7bc0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:45:05.337524 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 605/1221] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:45:05 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Aug 28 00:45:05.646080 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4a445b6-dfa8-4395-8638-e1703697ab7a None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:45:05.822433 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4a445b6-dfa8-4395-8638-e1703697ab7a None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:45:05.863018 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abd965ce-935b-4ea1-beaf-649b06965164 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:45:05.863771 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abd965ce-935b-4ea1-beaf-649b06965164 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:45:05.864139 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abd965ce-935b-4ea1-beaf-649b06965164 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:45:05.864482 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abd965ce-935b-4ea1-beaf-649b06965164 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:45:05.878368 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abd965ce-935b-4ea1-beaf-649b06965164 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 28 00:45:05.883749 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-abd965ce-935b-4ea1-beaf-649b06965164 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 28 00:45:05.883749 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:45:05.883749 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:45:05.883749 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:45:05.883749 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:45:05.883749 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:45:05.883749 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:45:05.883749 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:45:05.883749 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:45:05.883749 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:45:05.883749 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:45:05.883749 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:45:05.883749 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 28 00:45:05.883749 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 28 00:45:05.883749 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 28 00:45:05.883749 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 28 00:45:05.885749 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 28 00:45:05.885749 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 28 00:45:05.885749 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 28 00:45:05.885749 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 28 00:45:05.885749 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:45:05.885749 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:45:05.885749 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:45:05.885749 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:45:05.885749 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:45:05.885749 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:45:05.885749 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:45:05.885749 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:45:05.885749 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:45:05.885749 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:45:05.885749 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:45:05.885749 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:45:05.887346 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:45:05.887346 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:45:05.887346 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:45:05.887346 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:45:05.887346 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 28 00:45:05.887346 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 28 00:45:05.887346 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 28 00:45:05.887346 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 28 00:45:05.887346 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 28 00:45:05.887346 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 28 00:45:05.887346 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 28 00:45:05.887346 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:45:05.891932 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 617/1222] 10.222.0.42 () {56 vars in 1102 bytes} [Sun Aug 28 00:45:05 2022] GET /identity/v3/roles/service => generated 85 bytes in 247 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:45:05.894116 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bff86d36-1718-4f4c-af1f-3ead392d7bc0 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:45:06.036831 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bff86d36-1718-4f4c-af1f-3ead392d7bc0 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:45:06.063883 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:45:06.079025 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7a4dea6-7fe5-4791-ba8d-4a9bc64eec06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:45:06.079688 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7a4dea6-7fe5-4791-ba8d-4a9bc64eec06 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:45:06.080117 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7a4dea6-7fe5-4791-ba8d-4a9bc64eec06 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:45:06.080401 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7a4dea6-7fe5-4791-ba8d-4a9bc64eec06 None admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:45:06.081138 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7a4dea6-7fe5-4791-ba8d-4a9bc64eec06 None admin] RBAC: Adding query filter params (name=service) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:45:06.081519 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7a4dea6-7fe5-4791-ba8d-4a9bc64eec06 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:45:06.083093 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7a4dea6-7fe5-4791-ba8d-4a9bc64eec06 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:45:06.093845 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 606/1223] 10.222.0.42 () {56 vars in 1103 bytes} [Sun Aug 28 00:45:05 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:45:06.103175 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abd965ce-935b-4ea1-beaf-649b06965164 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:45:06.170914 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:45:06.251320 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abd965ce-935b-4ea1-beaf-649b06965164 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:45:06.295997 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b243e7d-40e0-4562-9626-828b9f3a4d7b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:45:06.296929 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b243e7d-40e0-4562-9626-828b9f3a4d7b None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:45:06.297447 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b243e7d-40e0-4562-9626-828b9f3a4d7b None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:45:06.297925 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b243e7d-40e0-4562-9626-828b9f3a4d7b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:45:06.318768 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b243e7d-40e0-4562-9626-828b9f3a4d7b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:45:06.320672 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b243e7d-40e0-4562-9626-828b9f3a4d7b None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:45:06.339319 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-4b243e7d-40e0-4562-9626-828b9f3a4d7b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:45:06.339319 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:45:06.339319 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:45:06.339319 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:45:06.339319 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:45:06.339319 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:45:06.339319 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:45:06.339319 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:45:06.339319 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:45:06.339319 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:45:06.339319 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:45:06.339319 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:45:06.339319 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 28 00:45:06.339319 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 28 00:45:06.339319 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 28 00:45:06.339319 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 28 00:45:06.341383 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:45:06.341383 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:45:06.341383 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:45:06.341383 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:45:06.341383 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:45:06.341383 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:45:06.341383 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:45:06.341383 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:45:06.341383 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:45:06.341383 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:45:06.341383 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:45:06.341383 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:45:06.341383 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:45:06.341383 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:45:06.341383 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:45:06.341383 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:45:06.341383 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 28 00:45:06.343065 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 28 00:45:06.343065 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:45:06.343065 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:45:06.343065 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 618/1224] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:45:06 2022] GET /identity/v3/domains/Default => generated 87 bytes in 238 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 28 00:45:06.349431 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7a4dea6-7fe5-4791-ba8d-4a9bc64eec06 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:45:06.467475 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7a4dea6-7fe5-4791-ba8d-4a9bc64eec06 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:45:06.511402 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1430afe6-c608-4cd4-9232-6a4a941530f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:45:06.513141 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1430afe6-c608-4cd4-9232-6a4a941530f7 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:45:06.513477 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1430afe6-c608-4cd4-9232-6a4a941530f7 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:45:06.513839 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1430afe6-c608-4cd4-9232-6a4a941530f7 None admin] PATH_INFO: `/v3/domains` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:45:06.514907 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1430afe6-c608-4cd4-9232-6a4a941530f7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:45:06.515399 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1430afe6-c608-4cd4-9232-6a4a941530f7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:45:06.517746 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1430afe6-c608-4cd4-9232-6a4a941530f7 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:45:06.543831 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 607/1225] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Aug 28 00:45:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:45:06.545877 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b243e7d-40e0-4562-9626-828b9f3a4d7b None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:45:06.689666 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:45:06.704509 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b243e7d-40e0-4562-9626-828b9f3a4d7b None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:45:06.771430 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50146cfc-4e77-458a-972e-bc24eb8d5032 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:45:06.772514 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50146cfc-4e77-458a-972e-bc24eb8d5032 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:45:06.773046 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50146cfc-4e77-458a-972e-bc24eb8d5032 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:45:06.773538 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50146cfc-4e77-458a-972e-bc24eb8d5032 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:45:06.791369 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50146cfc-4e77-458a-972e-bc24eb8d5032 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:45:06.794450 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50146cfc-4e77-458a-972e-bc24eb8d5032 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:45:06.811628 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-50146cfc-4e77-458a-972e-bc24eb8d5032 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Aug 28 00:45:06.811628 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:45:06.811628 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:45:06.811628 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:45:06.811628 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:45:06.811628 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:45:06.811628 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:45:06.811628 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:45:06.811628 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:45:06.811628 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:45:06.811628 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:45:06.811628 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:45:06.811628 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 28 00:45:06.811628 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 28 00:45:06.811628 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 28 00:45:06.811628 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 28 00:45:06.813784 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:45:06.813784 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:45:06.813784 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 28 00:45:06.813784 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 28 00:45:06.813784 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 28 00:45:06.813784 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 28 00:45:06.813784 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:45:06.813784 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:45:06.813784 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:45:06.813784 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:45:06.813784 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:45:06.813784 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:45:06.813784 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:45:06.813784 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:45:06.813784 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:45:06.813784 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:45:06.813784 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:45:06.813784 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:45:06.815981 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:45:06.815981 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:45:06.815981 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 28 00:45:06.815981 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 28 00:45:06.815981 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 28 00:45:06.815981 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 28 00:45:06.815981 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 28 00:45:06.815981 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 28 00:45:06.815981 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Aug 28 00:45:06.815981 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:45:06.815981 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 619/1226] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:45:06 2022] GET /identity/v3/users/placement => generated 87 bytes in 268 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:45:06.822845 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1430afe6-c608-4cd4-9232-6a4a941530f7 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:45:06.895721 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:45:06.974387 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1430afe6-c608-4cd4-9232-6a4a941530f7 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:45:07.031765 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b6011b7-c061-4794-bc32-c27881f5c00d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:45:07.032390 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b6011b7-c061-4794-bc32-c27881f5c00d None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:45:07.032675 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b6011b7-c061-4794-bc32-c27881f5c00d None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:45:07.032948 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b6011b7-c061-4794-bc32-c27881f5c00d None admin] PATH_INFO: `/v3/users` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:45:07.033722 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b6011b7-c061-4794-bc32-c27881f5c00d None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:45:07.034033 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b6011b7-c061-4794-bc32-c27881f5c00d None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:45:07.035649 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b6011b7-c061-4794-bc32-c27881f5c00d None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:45:07.058124 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 608/1227] 10.222.0.42 () {56 vars in 1143 bytes} [Sun Aug 28 00:45:06 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 237 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:45:07.064694 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50146cfc-4e77-458a-972e-bc24eb8d5032 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:45:07.214039 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50146cfc-4e77-458a-972e-bc24eb8d5032 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:45:07.260726 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5f7ee7b-6ab7-4b64-a822-a1ee19f46f49 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:45:07.261820 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5f7ee7b-6ab7-4b64-a822-a1ee19f46f49 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:45:07.262147 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5f7ee7b-6ab7-4b64-a822-a1ee19f46f49 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:45:07.262573 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5f7ee7b-6ab7-4b64-a822-a1ee19f46f49 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:45:07.281689 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5f7ee7b-6ab7-4b64-a822-a1ee19f46f49 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:45:07.283657 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5f7ee7b-6ab7-4b64-a822-a1ee19f46f49 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:45:07.306942 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-e5f7ee7b-6ab7-4b64-a822-a1ee19f46f49 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:45:07.306942 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:45:07.306942 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:45:07.306942 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:45:07.306942 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:45:07.306942 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:45:07.306942 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:45:07.306942 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:45:07.306942 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:45:07.306942 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:45:07.306942 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:45:07.306942 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:45:07.306942 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 28 00:45:07.306942 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 28 00:45:07.306942 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 28 00:45:07.306942 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 28 00:45:07.309273 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:45:07.309273 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:45:07.309273 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:45:07.309273 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:45:07.309273 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:45:07.309273 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:45:07.309273 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:45:07.309273 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:45:07.309273 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:45:07.309273 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:45:07.309273 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:45:07.309273 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:45:07.309273 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:45:07.309273 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:45:07.309273 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:45:07.309273 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:45:07.309273 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 28 00:45:07.311377 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 28 00:45:07.311377 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:45:07.311377 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:45:07.311377 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 620/1228] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:45:07 2022] GET /identity/v3/domains/Default => generated 87 bytes in 244 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 28 00:45:07.315973 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b6011b7-c061-4794-bc32-c27881f5c00d None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:45:07.471068 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b6011b7-c061-4794-bc32-c27881f5c00d None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:45:07.475461 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:45:07.525034 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7995bf10-1d70-49ac-ae5c-c16201251d69 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:45:07.525694 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7995bf10-1d70-49ac-ae5c-c16201251d69 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:45:07.526068 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7995bf10-1d70-49ac-ae5c-c16201251d69 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:45:07.526475 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7995bf10-1d70-49ac-ae5c-c16201251d69 None admin] PATH_INFO: `/v3/domains` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:45:07.527251 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7995bf10-1d70-49ac-ae5c-c16201251d69 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:45:07.527700 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7995bf10-1d70-49ac-ae5c-c16201251d69 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:45:07.529666 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7995bf10-1d70-49ac-ae5c-c16201251d69 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:45:07.547223 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 609/1229] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Aug 28 00:45:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:45:07.554181 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5f7ee7b-6ab7-4b64-a822-a1ee19f46f49 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:45:07.569008 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:45:07.700225 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5f7ee7b-6ab7-4b64-a822-a1ee19f46f49 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:45:07.747583 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f37dbcb-0505-4f97-b832-dc130b17abdc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:45:07.748449 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f37dbcb-0505-4f97-b832-dc130b17abdc None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:45:07.748826 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f37dbcb-0505-4f97-b832-dc130b17abdc None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:45:07.749304 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f37dbcb-0505-4f97-b832-dc130b17abdc None admin] PATH_INFO: `/v3/projects/service` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:45:07.763191 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f37dbcb-0505-4f97-b832-dc130b17abdc None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:45:07.765919 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f37dbcb-0505-4f97-b832-dc130b17abdc None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:45:07.785343 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-8f37dbcb-0505-4f97-b832-dc130b17abdc None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 28 00:45:07.785343 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:45:07.785343 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:45:07.785343 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:45:07.785343 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:45:07.785343 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:45:07.785343 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:45:07.785343 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:45:07.785343 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:45:07.785343 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:45:07.785343 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:45:07.785343 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:45:07.785343 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 28 00:45:07.785343 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 28 00:45:07.785343 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 28 00:45:07.787318 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 28 00:45:07.787318 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:45:07.787318 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:45:07.787318 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:45:07.787318 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:45:07.787318 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:45:07.787318 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:45:07.787318 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:45:07.787318 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:45:07.787318 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:45:07.787318 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:45:07.787318 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:45:07.787318 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:45:07.787318 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:45:07.787318 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:45:07.787318 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:45:07.787318 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:45:07.789249 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 28 00:45:07.789249 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 28 00:45:07.789249 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 28 00:45:07.789249 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 28 00:45:07.789249 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 28 00:45:07.789249 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 28 00:45:07.789249 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 28 00:45:07.789249 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:45:07.789249 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 621/1230] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Aug 28 00:45:07 2022] GET /identity/v3/projects/service => generated 88 bytes in 234 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:45:07.795041 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7995bf10-1d70-49ac-ae5c-c16201251d69 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:45:07.950740 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7995bf10-1d70-49ac-ae5c-c16201251d69 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:45:08.003136 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30148504-3570-44d2-a7dd-3890b4de7573 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:45:08.003955 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30148504-3570-44d2-a7dd-3890b4de7573 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:45:08.004451 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30148504-3570-44d2-a7dd-3890b4de7573 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:45:08.004953 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30148504-3570-44d2-a7dd-3890b4de7573 None admin] PATH_INFO: `/v3/projects` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:45:08.006280 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30148504-3570-44d2-a7dd-3890b4de7573 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:45:08.006713 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30148504-3570-44d2-a7dd-3890b4de7573 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:45:08.008892 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30148504-3570-44d2-a7dd-3890b4de7573 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:45:08.028522 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 610/1231] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Aug 28 00:45:07 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 234 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:45:08.036209 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f37dbcb-0505-4f97-b832-dc130b17abdc None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:45:08.160637 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:45:08.224402 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f37dbcb-0505-4f97-b832-dc130b17abdc None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:45:08.282727 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee40094e-ee12-4da1-b985-4d2c5792a553 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:45:08.283475 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee40094e-ee12-4da1-b985-4d2c5792a553 None admin] REQUEST_METHOD: `PUT` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:45:08.283665 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee40094e-ee12-4da1-b985-4d2c5792a553 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:45:08.283934 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee40094e-ee12-4da1-b985-4d2c5792a553 None admin] PATH_INFO: `/v3/projects/f53e6fdb369b480fbebb4086a505df67/users/c2a3834fa6894dada3291e4781bbd553/roles/348070fea3dd4cc2a6e00e29fe82cc25` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:45:08.303975 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee40094e-ee12-4da1-b985-4d2c5792a553 None admin] RBAC: Authorizing `identity:create_grant(project_id=f53e6fdb369b480fbebb4086a505df67, user_id=c2a3834fa6894dada3291e4781bbd553, role_id=348070fea3dd4cc2a6e00e29fe82cc25)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:45:08.306796 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee40094e-ee12-4da1-b985-4d2c5792a553 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:45:08.327740 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 622/1232] 10.222.0.42 () {58 vars in 1439 bytes} [Sun Aug 28 00:45:08 2022] PUT /identity/v3/projects/f53e6fdb369b480fbebb4086a505df67/users/c2a3834fa6894dada3291e4781bbd553/roles/348070fea3dd4cc2a6e00e29fe82cc25 => generated 0 bytes in 292 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 28 00:45:10.516890 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbf1f7fa-b59a-4333-9322-8748b9058785 None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:45:10.517834 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbf1f7fa-b59a-4333-9322-8748b9058785 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:45:10.517834 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbf1f7fa-b59a-4333-9322-8748b9058785 None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:45:10.519463 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 611/1233] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:45:10 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:45:10.525837 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ba7e6c6-179d-4717-8870-4594342fd14e None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:45:10.526576 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ba7e6c6-179d-4717-8870-4594342fd14e None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:45:10.526576 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ba7e6c6-179d-4717-8870-4594342fd14e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:45:10.547996 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-8ba7e6c6-179d-4717-8870-4594342fd14e None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:45:10.549380 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-8ba7e6c6-179d-4717-8870-4594342fd14e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:45:10.630752 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 623/1234] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:45:10 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 28 00:45:10.917651 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbf1f7fa-b59a-4333-9322-8748b9058785 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:45:10.959532 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:45:11.071162 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbf1f7fa-b59a-4333-9322-8748b9058785 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:45:11.121095 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70370eb6-61c4-47da-b5b0-62b4a323fa37 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:45:11.122206 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70370eb6-61c4-47da-b5b0-62b4a323fa37 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:45:11.122633 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70370eb6-61c4-47da-b5b0-62b4a323fa37 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:45:11.123150 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70370eb6-61c4-47da-b5b0-62b4a323fa37 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:45:11.136365 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70370eb6-61c4-47da-b5b0-62b4a323fa37 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 28 00:45:11.141936 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-70370eb6-61c4-47da-b5b0-62b4a323fa37 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Aug 28 00:45:11.141936 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:45:11.141936 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:45:11.141936 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:45:11.141936 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:45:11.141936 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:45:11.141936 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:45:11.141936 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:45:11.141936 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:45:11.141936 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:45:11.141936 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:45:11.141936 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:45:11.141936 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 28 00:45:11.141936 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 28 00:45:11.141936 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 28 00:45:11.141936 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 28 00:45:11.143850 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 28 00:45:11.143850 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 28 00:45:11.143850 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 28 00:45:11.143850 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 28 00:45:11.143850 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:45:11.143850 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:45:11.143850 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:45:11.143850 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:45:11.143850 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:45:11.143850 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:45:11.143850 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:45:11.143850 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:45:11.143850 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:45:11.143850 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:45:11.143850 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:45:11.143850 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:45:11.145536 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:45:11.145536 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:45:11.145536 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:45:11.145536 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:45:11.145536 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 28 00:45:11.145536 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 28 00:45:11.145536 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 28 00:45:11.145536 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 28 00:45:11.145536 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 28 00:45:11.145536 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 28 00:45:11.145536 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Aug 28 00:45:11.145536 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:45:11.145536 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 612/1235] 10.222.0.42 () {56 vars in 1102 bytes} [Sun Aug 28 00:45:10 2022] GET /identity/v3/roles/service => generated 85 bytes in 227 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:45:11.153519 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ba7e6c6-179d-4717-8870-4594342fd14e None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:45:11.222771 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:45:11.300283 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ba7e6c6-179d-4717-8870-4594342fd14e None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:45:11.359206 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa382f94-685f-433b-8820-0286fb4aba8a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:45:11.359877 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa382f94-685f-433b-8820-0286fb4aba8a None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:45:11.360155 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa382f94-685f-433b-8820-0286fb4aba8a None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:45:11.360429 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa382f94-685f-433b-8820-0286fb4aba8a None admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:45:11.361209 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa382f94-685f-433b-8820-0286fb4aba8a None admin] RBAC: Adding query filter params (name=service) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:45:11.361532 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa382f94-685f-433b-8820-0286fb4aba8a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:45:11.363065 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa382f94-685f-433b-8820-0286fb4aba8a None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:45:11.374143 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 624/1236] 10.222.0.42 () {56 vars in 1103 bytes} [Sun Aug 28 00:45:11 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:45:11.380430 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70370eb6-61c4-47da-b5b0-62b4a323fa37 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:45:11.505142 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:45:11.535391 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70370eb6-61c4-47da-b5b0-62b4a323fa37 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:45:11.596144 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-293c73c1-6cb0-4453-8099-498dfbed7ff6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:45:11.597276 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-293c73c1-6cb0-4453-8099-498dfbed7ff6 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:45:11.597773 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-293c73c1-6cb0-4453-8099-498dfbed7ff6 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:45:11.598191 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-293c73c1-6cb0-4453-8099-498dfbed7ff6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:45:11.611865 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-293c73c1-6cb0-4453-8099-498dfbed7ff6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:45:11.613658 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-293c73c1-6cb0-4453-8099-498dfbed7ff6 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:45:11.635193 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-293c73c1-6cb0-4453-8099-498dfbed7ff6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:45:11.635193 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:45:11.635193 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:45:11.635193 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:45:11.635193 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:45:11.635193 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:45:11.635193 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:45:11.635193 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:45:11.635193 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:45:11.635193 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:45:11.635193 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:45:11.635193 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:45:11.635193 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 28 00:45:11.635193 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 28 00:45:11.635193 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 28 00:45:11.635193 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 28 00:45:11.637311 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:45:11.637311 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:45:11.637311 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:45:11.637311 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:45:11.637311 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:45:11.637311 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:45:11.637311 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:45:11.637311 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:45:11.637311 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:45:11.637311 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:45:11.637311 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:45:11.637311 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:45:11.637311 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:45:11.637311 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:45:11.637311 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:45:11.637311 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:45:11.637311 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 28 00:45:11.638464 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 28 00:45:11.638464 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:45:11.638464 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:45:11.638464 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 613/1237] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:45:11 2022] GET /identity/v3/domains/Default => generated 87 bytes in 258 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 28 00:45:11.646753 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa382f94-685f-433b-8820-0286fb4aba8a None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:45:11.787798 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa382f94-685f-433b-8820-0286fb4aba8a None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:45:11.800716 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:45:11.845283 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a6ae82a-664c-49e5-a6df-c6cdd7752202 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:45:11.846064 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a6ae82a-664c-49e5-a6df-c6cdd7752202 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:45:11.846353 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a6ae82a-664c-49e5-a6df-c6cdd7752202 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:45:11.846652 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a6ae82a-664c-49e5-a6df-c6cdd7752202 None admin] PATH_INFO: `/v3/domains` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:45:11.847340 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a6ae82a-664c-49e5-a6df-c6cdd7752202 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:45:11.847641 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a6ae82a-664c-49e5-a6df-c6cdd7752202 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:45:11.849295 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a6ae82a-664c-49e5-a6df-c6cdd7752202 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:45:11.865937 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 625/1238] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Aug 28 00:45:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:45:11.874951 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-293c73c1-6cb0-4453-8099-498dfbed7ff6 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:45:12.032089 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-293c73c1-6cb0-4453-8099-498dfbed7ff6 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:45:12.087784 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b99c2c39-343f-4cb5-b8c2-fd015466d2f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:45:12.088532 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b99c2c39-343f-4cb5-b8c2-fd015466d2f6 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:45:12.089017 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b99c2c39-343f-4cb5-b8c2-fd015466d2f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:45:12.089476 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b99c2c39-343f-4cb5-b8c2-fd015466d2f6 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:45:12.093973 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:45:12.102875 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b99c2c39-343f-4cb5-b8c2-fd015466d2f6 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:45:12.105582 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b99c2c39-343f-4cb5-b8c2-fd015466d2f6 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:45:12.126355 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-b99c2c39-343f-4cb5-b8c2-fd015466d2f6 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Aug 28 00:45:12.126355 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:45:12.126355 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:45:12.126355 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:45:12.126355 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:45:12.126355 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:45:12.126355 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:45:12.126355 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:45:12.126355 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:45:12.126355 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:45:12.126355 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:45:12.126355 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:45:12.126355 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 28 00:45:12.126355 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 28 00:45:12.126355 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 28 00:45:12.126355 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 28 00:45:12.127882 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:45:12.127882 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:45:12.127882 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 28 00:45:12.127882 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 28 00:45:12.127882 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 28 00:45:12.127882 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 28 00:45:12.127882 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:45:12.127882 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:45:12.127882 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:45:12.127882 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:45:12.127882 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:45:12.127882 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:45:12.127882 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:45:12.127882 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:45:12.127882 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:45:12.127882 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:45:12.127882 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:45:12.127882 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:45:12.129936 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:45:12.129936 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:45:12.129936 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 28 00:45:12.129936 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 28 00:45:12.129936 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 28 00:45:12.129936 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 28 00:45:12.129936 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 28 00:45:12.129936 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 28 00:45:12.129936 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Aug 28 00:45:12.129936 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:45:12.129936 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 614/1239] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:45:11 2022] GET /identity/v3/users/placement => generated 87 bytes in 254 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 28 00:45:12.136763 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a6ae82a-664c-49e5-a6df-c6cdd7752202 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:45:12.296475 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a6ae82a-664c-49e5-a6df-c6cdd7752202 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:45:12.357841 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec03d145-25ec-4476-a219-936a02eddcf5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:45:12.358698 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec03d145-25ec-4476-a219-936a02eddcf5 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:45:12.358987 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec03d145-25ec-4476-a219-936a02eddcf5 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:45:12.359480 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec03d145-25ec-4476-a219-936a02eddcf5 None admin] PATH_INFO: `/v3/users` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:45:12.366266 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec03d145-25ec-4476-a219-936a02eddcf5 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:45:12.366832 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec03d145-25ec-4476-a219-936a02eddcf5 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:45:12.369859 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec03d145-25ec-4476-a219-936a02eddcf5 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:45:12.403945 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 626/1240] 10.222.0.42 () {56 vars in 1143 bytes} [Sun Aug 28 00:45:12 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 268 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:45:12.411358 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b99c2c39-343f-4cb5-b8c2-fd015466d2f6 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:45:12.566223 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b99c2c39-343f-4cb5-b8c2-fd015466d2f6 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:45:12.613942 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98521f59-cdb5-4fc5-aa11-2a00ac180a61 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:45:12.614465 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98521f59-cdb5-4fc5-aa11-2a00ac180a61 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:45:12.614744 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98521f59-cdb5-4fc5-aa11-2a00ac180a61 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:45:12.615019 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98521f59-cdb5-4fc5-aa11-2a00ac180a61 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:45:12.712459 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:45:12.721839 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98521f59-cdb5-4fc5-aa11-2a00ac180a61 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:45:12.723589 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98521f59-cdb5-4fc5-aa11-2a00ac180a61 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:45:12.847049 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 615/1241] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Aug 28 00:45:12 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 436 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 28 00:45:12.853842 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec03d145-25ec-4476-a219-936a02eddcf5 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:45:12.892114 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:45:12.999928 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec03d145-25ec-4476-a219-936a02eddcf5 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:45:13.055600 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b7ee9f6-16a9-491f-8a6d-c75eab54239a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:45:13.056658 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b7ee9f6-16a9-491f-8a6d-c75eab54239a None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:45:13.057043 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b7ee9f6-16a9-491f-8a6d-c75eab54239a None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:45:13.057542 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b7ee9f6-16a9-491f-8a6d-c75eab54239a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:45:13.074062 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b7ee9f6-16a9-491f-8a6d-c75eab54239a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:45:13.075855 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b7ee9f6-16a9-491f-8a6d-c75eab54239a None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:45:13.098732 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-5b7ee9f6-16a9-491f-8a6d-c75eab54239a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:45:13.098732 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:45:13.098732 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:45:13.098732 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:45:13.098732 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:45:13.098732 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:45:13.098732 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:45:13.098732 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:45:13.098732 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:45:13.098732 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:45:13.098732 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:45:13.098732 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:45:13.098732 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 28 00:45:13.098732 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 28 00:45:13.098732 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 28 00:45:13.098732 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 28 00:45:13.099981 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:45:13.099981 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:45:13.099981 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:45:13.099981 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:45:13.099981 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:45:13.099981 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:45:13.099981 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:45:13.099981 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:45:13.099981 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:45:13.099981 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:45:13.099981 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:45:13.099981 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:45:13.099981 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:45:13.099981 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:45:13.099981 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:45:13.099981 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:45:13.099981 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 28 00:45:13.101208 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 28 00:45:13.101208 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:45:13.101208 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:45:13.101208 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 627/1242] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:45:12 2022] GET /identity/v3/domains/Default => generated 87 bytes in 247 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:45:13.106554 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98521f59-cdb5-4fc5-aa11-2a00ac180a61 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:45:13.221760 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:45:13.231655 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98521f59-cdb5-4fc5-aa11-2a00ac180a61 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:45:13.278507 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-090bf4c8-9504-4252-83ae-3bfd8c71365d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:45:13.279136 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-090bf4c8-9504-4252-83ae-3bfd8c71365d None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:45:13.279408 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-090bf4c8-9504-4252-83ae-3bfd8c71365d None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:45:13.279670 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-090bf4c8-9504-4252-83ae-3bfd8c71365d None admin] PATH_INFO: `/v3/domains` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:45:13.280316 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-090bf4c8-9504-4252-83ae-3bfd8c71365d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:45:13.280613 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-090bf4c8-9504-4252-83ae-3bfd8c71365d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:45:13.282185 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-090bf4c8-9504-4252-83ae-3bfd8c71365d None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:45:13.297388 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 616/1243] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Aug 28 00:45:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 28 00:45:13.305971 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b7ee9f6-16a9-491f-8a6d-c75eab54239a None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:45:13.404260 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:45:13.437174 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b7ee9f6-16a9-491f-8a6d-c75eab54239a None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:45:13.483882 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5381a24b-14a9-4804-aafa-893e999a734c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:45:13.484413 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5381a24b-14a9-4804-aafa-893e999a734c None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:45:13.484688 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5381a24b-14a9-4804-aafa-893e999a734c None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:45:13.485027 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5381a24b-14a9-4804-aafa-893e999a734c None admin] PATH_INFO: `/v3/projects/service` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:45:13.498778 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5381a24b-14a9-4804-aafa-893e999a734c None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:45:13.501567 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5381a24b-14a9-4804-aafa-893e999a734c None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:45:13.523887 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-5381a24b-14a9-4804-aafa-893e999a734c None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 28 00:45:13.523887 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:45:13.523887 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:45:13.523887 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:45:13.523887 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:45:13.523887 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:45:13.523887 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:45:13.523887 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:45:13.523887 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:45:13.523887 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:45:13.523887 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:45:13.523887 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:45:13.523887 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 28 00:45:13.523887 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 28 00:45:13.523887 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 28 00:45:13.524946 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 28 00:45:13.524946 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:45:13.524946 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:45:13.524946 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:45:13.524946 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:45:13.524946 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:45:13.524946 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:45:13.524946 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:45:13.524946 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:45:13.524946 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:45:13.524946 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:45:13.524946 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:45:13.524946 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:45:13.524946 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:45:13.524946 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:45:13.524946 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:45:13.524946 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:45:13.525911 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 28 00:45:13.525911 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 28 00:45:13.525911 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 28 00:45:13.525911 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 28 00:45:13.525911 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 28 00:45:13.525911 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 28 00:45:13.525911 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 28 00:45:13.525911 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:45:13.526594 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 628/1244] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Aug 28 00:45:13 2022] GET /identity/v3/projects/service => generated 88 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:45:13.534642 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-090bf4c8-9504-4252-83ae-3bfd8c71365d None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:45:13.680849 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-090bf4c8-9504-4252-83ae-3bfd8c71365d None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:45:13.720119 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8c4c624-b40d-4fb3-98ff-cabc7e3c2731 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:45:13.721123 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8c4c624-b40d-4fb3-98ff-cabc7e3c2731 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:45:13.721514 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8c4c624-b40d-4fb3-98ff-cabc7e3c2731 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:45:13.721860 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8c4c624-b40d-4fb3-98ff-cabc7e3c2731 None admin] PATH_INFO: `/v3/projects` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:45:13.722939 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8c4c624-b40d-4fb3-98ff-cabc7e3c2731 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:45:13.723389 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8c4c624-b40d-4fb3-98ff-cabc7e3c2731 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:45:13.725694 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8c4c624-b40d-4fb3-98ff-cabc7e3c2731 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:45:13.743446 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 617/1245] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Aug 28 00:45:13 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 28 00:45:13.748424 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5381a24b-14a9-4804-aafa-893e999a734c None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:45:13.875423 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5381a24b-14a9-4804-aafa-893e999a734c None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:45:13.920022 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3025dfc9-83a0-4322-b512-b42d6705a156 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:45:13.920650 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3025dfc9-83a0-4322-b512-b42d6705a156 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:45:13.920936 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3025dfc9-83a0-4322-b512-b42d6705a156 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:45:13.921262 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3025dfc9-83a0-4322-b512-b42d6705a156 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:45:13.921994 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3025dfc9-83a0-4322-b512-b42d6705a156 None admin] RBAC: Adding query filter params (role.id=348070fea3dd4cc2a6e00e29fe82cc25, scope.project.id=f53e6fdb369b480fbebb4086a505df67, user.id=c2a3834fa6894dada3291e4781bbd553) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:45:13.922296 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3025dfc9-83a0-4322-b512-b42d6705a156 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:45:13.923854 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3025dfc9-83a0-4322-b512-b42d6705a156 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:45:13.931574 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 629/1246] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Aug 28 00:45:13 2022] GET /identity/v3/role_assignments?user.id=c2a3834fa6894dada3291e4781bbd553&scope.project.id=f53e6fdb369b480fbebb4086a505df67&role.id=348070fea3dd4cc2a6e00e29fe82cc25 => generated 608 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:45:16.008790 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6f53083-1be9-44d0-b055-f5975486c592 None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:45:16.009488 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6f53083-1be9-44d0-b055-f5975486c592 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:45:16.009488 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6f53083-1be9-44d0-b055-f5975486c592 None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:45:16.011050 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 618/1247] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:45:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:45:16.017732 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f6d733b-2c08-4fff-ab2f-25c0a587cd6b None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:45:16.018398 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f6d733b-2c08-4fff-ab2f-25c0a587cd6b None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:45:16.018398 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f6d733b-2c08-4fff-ab2f-25c0a587cd6b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:45:16.027137 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:45:16.046684 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-0f6d733b-2c08-4fff-ab2f-25c0a587cd6b None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:45:16.048052 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-0f6d733b-2c08-4fff-ab2f-25c0a587cd6b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:45:16.144457 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 630/1248] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:45:16 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 28 00:45:16.436539 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6f53083-1be9-44d0-b055-f5975486c592 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:45:16.458593 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:45:16.568309 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6f53083-1be9-44d0-b055-f5975486c592 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:45:16.624353 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8ae5c72-e766-4545-a948-4f93af57a796 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:45:16.625089 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8ae5c72-e766-4545-a948-4f93af57a796 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:45:16.631647 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8ae5c72-e766-4545-a948-4f93af57a796 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:45:16.631647 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8ae5c72-e766-4545-a948-4f93af57a796 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:45:16.639131 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8ae5c72-e766-4545-a948-4f93af57a796 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 28 00:45:16.645881 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-d8ae5c72-e766-4545-a948-4f93af57a796 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 28 00:45:16.645881 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:45:16.645881 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:45:16.645881 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:45:16.645881 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:45:16.645881 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:45:16.645881 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:45:16.645881 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:45:16.645881 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:45:16.645881 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:45:16.645881 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:45:16.645881 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:45:16.645881 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 28 00:45:16.645881 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 28 00:45:16.645881 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 28 00:45:16.645881 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 28 00:45:16.648489 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 28 00:45:16.648489 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 28 00:45:16.648489 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 28 00:45:16.648489 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 28 00:45:16.648489 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:45:16.648489 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:45:16.648489 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:45:16.648489 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:45:16.648489 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:45:16.648489 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:45:16.648489 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:45:16.648489 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:45:16.648489 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:45:16.648489 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:45:16.648489 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:45:16.648489 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:45:16.650537 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:45:16.650537 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:45:16.650537 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:45:16.650537 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:45:16.650537 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 28 00:45:16.650537 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 28 00:45:16.650537 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 28 00:45:16.650537 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 28 00:45:16.650537 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 28 00:45:16.650537 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 28 00:45:16.650537 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 28 00:45:16.650537 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:45:16.650537 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 619/1249] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Aug 28 00:45:16 2022] GET /identity/v3/roles/admin => generated 83 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 28 00:45:16.656524 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f6d733b-2c08-4fff-ab2f-25c0a587cd6b None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:45:16.818692 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f6d733b-2c08-4fff-ab2f-25c0a587cd6b None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:45:16.866888 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:45:16.874851 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3164e86-7857-4a35-815c-e1eca66b482e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:45:16.875595 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3164e86-7857-4a35-815c-e1eca66b482e None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:45:16.875993 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3164e86-7857-4a35-815c-e1eca66b482e None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:45:16.876433 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3164e86-7857-4a35-815c-e1eca66b482e None admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:45:16.877674 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3164e86-7857-4a35-815c-e1eca66b482e None admin] RBAC: Adding query filter params (name=admin) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:45:16.878161 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3164e86-7857-4a35-815c-e1eca66b482e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:45:16.880122 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3164e86-7857-4a35-815c-e1eca66b482e None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:45:16.897724 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 631/1250] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Aug 28 00:45:16 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 242 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:45:16.905411 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8ae5c72-e766-4545-a948-4f93af57a796 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:45:17.031009 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:45:17.069140 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8ae5c72-e766-4545-a948-4f93af57a796 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:45:17.123864 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-907fb7fd-961f-4c3e-8a6d-ad8f0fea2f09 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:45:17.124776 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907fb7fd-961f-4c3e-8a6d-ad8f0fea2f09 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:45:17.125242 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907fb7fd-961f-4c3e-8a6d-ad8f0fea2f09 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:45:17.125693 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907fb7fd-961f-4c3e-8a6d-ad8f0fea2f09 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:45:17.139395 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-907fb7fd-961f-4c3e-8a6d-ad8f0fea2f09 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:45:17.141553 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-907fb7fd-961f-4c3e-8a6d-ad8f0fea2f09 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:45:17.163937 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-907fb7fd-961f-4c3e-8a6d-ad8f0fea2f09 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:45:17.163937 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:45:17.163937 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:45:17.163937 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:45:17.163937 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:45:17.163937 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:45:17.163937 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:45:17.163937 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:45:17.163937 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:45:17.163937 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:45:17.163937 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:45:17.163937 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:45:17.163937 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 28 00:45:17.163937 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 28 00:45:17.163937 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 28 00:45:17.163937 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 28 00:45:17.173417 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:45:17.173417 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:45:17.173417 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:45:17.173417 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:45:17.173417 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:45:17.173417 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:45:17.173417 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:45:17.173417 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:45:17.173417 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:45:17.173417 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:45:17.173417 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:45:17.173417 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:45:17.173417 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:45:17.173417 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:45:17.173417 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:45:17.173417 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:45:17.173417 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 28 00:45:17.174708 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 28 00:45:17.174708 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:45:17.174708 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:45:17.174708 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 620/1251] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:45:16 2022] GET /identity/v3/domains/Default => generated 87 bytes in 261 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:45:17.175370 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3164e86-7857-4a35-815c-e1eca66b482e None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:45:17.326632 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3164e86-7857-4a35-815c-e1eca66b482e None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:45:17.390774 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c900ae49-41b1-417e-a9d3-8c8e677685f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:45:17.391818 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c900ae49-41b1-417e-a9d3-8c8e677685f2 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:45:17.392339 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c900ae49-41b1-417e-a9d3-8c8e677685f2 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:45:17.392837 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c900ae49-41b1-417e-a9d3-8c8e677685f2 None admin] PATH_INFO: `/v3/domains` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:45:17.394069 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c900ae49-41b1-417e-a9d3-8c8e677685f2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:45:17.394624 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c900ae49-41b1-417e-a9d3-8c8e677685f2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:45:17.397120 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c900ae49-41b1-417e-a9d3-8c8e677685f2 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:45:17.418749 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 632/1252] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Aug 28 00:45:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 245 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:45:17.428113 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-907fb7fd-961f-4c3e-8a6d-ad8f0fea2f09 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:45:17.579524 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-907fb7fd-961f-4c3e-8a6d-ad8f0fea2f09 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:45:17.623114 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:45:17.638043 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b4f3c95-0e3f-46b2-b986-84df47b0b423 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:45:17.638830 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b4f3c95-0e3f-46b2-b986-84df47b0b423 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:45:17.639286 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b4f3c95-0e3f-46b2-b986-84df47b0b423 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:45:17.639749 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b4f3c95-0e3f-46b2-b986-84df47b0b423 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:45:17.651531 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b4f3c95-0e3f-46b2-b986-84df47b0b423 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:45:17.653743 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b4f3c95-0e3f-46b2-b986-84df47b0b423 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:45:17.675067 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-2b4f3c95-0e3f-46b2-b986-84df47b0b423 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Aug 28 00:45:17.675067 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:45:17.675067 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:45:17.675067 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:45:17.675067 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:45:17.675067 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:45:17.675067 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:45:17.675067 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:45:17.675067 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:45:17.675067 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:45:17.675067 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:45:17.675067 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:45:17.675067 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 28 00:45:17.675067 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 28 00:45:17.675067 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 28 00:45:17.675067 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 28 00:45:17.676931 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:45:17.676931 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:45:17.676931 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 28 00:45:17.676931 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 28 00:45:17.676931 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 28 00:45:17.676931 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 28 00:45:17.676931 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:45:17.676931 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:45:17.676931 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:45:17.676931 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:45:17.676931 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:45:17.676931 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:45:17.676931 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:45:17.676931 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:45:17.676931 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:45:17.676931 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:45:17.676931 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:45:17.676931 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:45:17.679275 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:45:17.679275 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:45:17.679275 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 28 00:45:17.679275 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 28 00:45:17.679275 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 28 00:45:17.679275 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 28 00:45:17.679275 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 28 00:45:17.679275 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 28 00:45:17.679275 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Aug 28 00:45:17.679275 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:45:17.679275 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 621/1253] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:45:17 2022] GET /identity/v3/users/placement => generated 87 bytes in 250 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:45:17.687164 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c900ae49-41b1-417e-a9d3-8c8e677685f2 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:45:17.777073 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:45:17.849418 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c900ae49-41b1-417e-a9d3-8c8e677685f2 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:45:17.911009 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f06e038-e48f-40e6-84f4-3ba3d652e17f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:45:17.912000 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f06e038-e48f-40e6-84f4-3ba3d652e17f None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:45:17.912458 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f06e038-e48f-40e6-84f4-3ba3d652e17f None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:45:17.912940 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f06e038-e48f-40e6-84f4-3ba3d652e17f None admin] PATH_INFO: `/v3/users` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:45:17.914454 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f06e038-e48f-40e6-84f4-3ba3d652e17f None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:45:17.915013 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f06e038-e48f-40e6-84f4-3ba3d652e17f None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:45:17.917603 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f06e038-e48f-40e6-84f4-3ba3d652e17f None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:45:17.945612 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 633/1254] 10.222.0.42 () {56 vars in 1143 bytes} [Sun Aug 28 00:45:17 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 259 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:45:17.952617 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b4f3c95-0e3f-46b2-b986-84df47b0b423 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:45:18.111724 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b4f3c95-0e3f-46b2-b986-84df47b0b423 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:45:18.157942 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79c27c84-ed42-4a1b-ab9d-0c85f6bf7763 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:45:18.158597 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79c27c84-ed42-4a1b-ab9d-0c85f6bf7763 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:45:18.158949 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79c27c84-ed42-4a1b-ab9d-0c85f6bf7763 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:45:18.159262 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79c27c84-ed42-4a1b-ab9d-0c85f6bf7763 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:45:18.233494 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:45:18.249176 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79c27c84-ed42-4a1b-ab9d-0c85f6bf7763 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:45:18.250658 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79c27c84-ed42-4a1b-ab9d-0c85f6bf7763 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:45:18.372595 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 622/1255] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Aug 28 00:45:17 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 421 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 28 00:45:18.378737 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f06e038-e48f-40e6-84f4-3ba3d652e17f None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:45:18.517894 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f06e038-e48f-40e6-84f4-3ba3d652e17f None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:45:18.535329 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:45:18.566424 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b06bc54d-1a70-4a3e-82e1-f20019f712a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:45:18.567384 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b06bc54d-1a70-4a3e-82e1-f20019f712a6 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:45:18.567816 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b06bc54d-1a70-4a3e-82e1-f20019f712a6 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:45:18.568319 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b06bc54d-1a70-4a3e-82e1-f20019f712a6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:45:18.587354 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b06bc54d-1a70-4a3e-82e1-f20019f712a6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:45:18.590183 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b06bc54d-1a70-4a3e-82e1-f20019f712a6 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:45:18.607465 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-b06bc54d-1a70-4a3e-82e1-f20019f712a6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:45:18.607465 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:45:18.607465 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:45:18.607465 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:45:18.607465 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:45:18.607465 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:45:18.607465 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:45:18.607465 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:45:18.607465 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:45:18.607465 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:45:18.607465 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:45:18.607465 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:45:18.607465 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 28 00:45:18.607465 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 28 00:45:18.607465 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 28 00:45:18.607465 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 28 00:45:18.609550 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:45:18.609550 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:45:18.609550 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:45:18.609550 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:45:18.609550 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:45:18.609550 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:45:18.609550 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:45:18.609550 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:45:18.609550 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:45:18.609550 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:45:18.609550 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:45:18.609550 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:45:18.609550 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:45:18.609550 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:45:18.609550 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:45:18.609550 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:45:18.609550 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 28 00:45:18.611302 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 28 00:45:18.611302 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:45:18.611302 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:45:18.611302 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 634/1256] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:45:18 2022] GET /identity/v3/domains/Default => generated 87 bytes in 231 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 28 00:45:18.614592 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79c27c84-ed42-4a1b-ab9d-0c85f6bf7763 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:45:18.734267 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:45:18.751111 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79c27c84-ed42-4a1b-ab9d-0c85f6bf7763 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:45:18.801942 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3563bd46-7f2f-4398-bf3b-4042385b7a99 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:45:18.803423 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3563bd46-7f2f-4398-bf3b-4042385b7a99 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:45:18.804007 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3563bd46-7f2f-4398-bf3b-4042385b7a99 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:45:18.804576 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3563bd46-7f2f-4398-bf3b-4042385b7a99 None admin] PATH_INFO: `/v3/domains` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:45:18.806005 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3563bd46-7f2f-4398-bf3b-4042385b7a99 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:45:18.806533 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3563bd46-7f2f-4398-bf3b-4042385b7a99 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:45:18.809318 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3563bd46-7f2f-4398-bf3b-4042385b7a99 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:45:18.825437 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 623/1257] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Aug 28 00:45:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 28 00:45:18.833021 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b06bc54d-1a70-4a3e-82e1-f20019f712a6 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:45:18.963191 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b06bc54d-1a70-4a3e-82e1-f20019f712a6 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:45:19.015128 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f68f5999-5d45-4492-b2fe-4511130d20e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:45:19.015851 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f68f5999-5d45-4492-b2fe-4511130d20e9 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:45:19.016180 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f68f5999-5d45-4492-b2fe-4511130d20e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:45:19.016520 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f68f5999-5d45-4492-b2fe-4511130d20e9 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:45:19.024678 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f68f5999-5d45-4492-b2fe-4511130d20e9 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:45:19.026519 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f68f5999-5d45-4492-b2fe-4511130d20e9 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:45:19.041687 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-f68f5999-5d45-4492-b2fe-4511130d20e9 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 28 00:45:19.041687 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:45:19.041687 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:45:19.041687 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:45:19.041687 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:45:19.041687 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:45:19.041687 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:45:19.041687 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:45:19.041687 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:45:19.041687 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:45:19.041687 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:45:19.041687 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:45:19.041687 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 28 00:45:19.041687 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 28 00:45:19.041687 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 28 00:45:19.043018 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 28 00:45:19.043018 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:45:19.043018 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:45:19.043018 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:45:19.043018 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:45:19.043018 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:45:19.043018 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:45:19.043018 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:45:19.043018 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:45:19.043018 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:45:19.043018 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:45:19.043018 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:45:19.043018 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:45:19.043018 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:45:19.043018 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:45:19.043018 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:45:19.043018 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:45:19.044034 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 28 00:45:19.044034 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 28 00:45:19.044034 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 28 00:45:19.044034 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 28 00:45:19.044034 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 28 00:45:19.044034 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 28 00:45:19.044034 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 28 00:45:19.044034 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:45:19.044034 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 635/1258] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Aug 28 00:45:18 2022] GET /identity/v3/projects/service => generated 88 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:45:19.052022 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3563bd46-7f2f-4398-bf3b-4042385b7a99 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:45:19.196856 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3563bd46-7f2f-4398-bf3b-4042385b7a99 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:45:19.257813 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-208e710d-2813-47cd-a138-1246388217cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:45:19.258839 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-208e710d-2813-47cd-a138-1246388217cd None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:45:19.259322 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-208e710d-2813-47cd-a138-1246388217cd None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:45:19.259789 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-208e710d-2813-47cd-a138-1246388217cd None admin] PATH_INFO: `/v3/projects` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:45:19.261011 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-208e710d-2813-47cd-a138-1246388217cd None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:45:19.261439 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-208e710d-2813-47cd-a138-1246388217cd None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:45:19.264025 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-208e710d-2813-47cd-a138-1246388217cd None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:45:19.280474 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 624/1259] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Aug 28 00:45:19 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 229 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 28 00:45:19.287533 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f68f5999-5d45-4492-b2fe-4511130d20e9 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:45:19.334108 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:45:19.440984 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f68f5999-5d45-4492-b2fe-4511130d20e9 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:45:19.504120 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa8a7d23-1d84-4688-a794-b78b94668466 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:45:19.505197 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa8a7d23-1d84-4688-a794-b78b94668466 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:45:19.505686 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa8a7d23-1d84-4688-a794-b78b94668466 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:45:19.506184 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa8a7d23-1d84-4688-a794-b78b94668466 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:45:19.507679 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa8a7d23-1d84-4688-a794-b78b94668466 None admin] RBAC: Adding query filter params (role.id=f08d6043261747e3b4763c744eef7dfb, scope.project.id=f53e6fdb369b480fbebb4086a505df67, user.id=c2a3834fa6894dada3291e4781bbd553) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:45:19.508308 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa8a7d23-1d84-4688-a794-b78b94668466 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:45:19.511487 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa8a7d23-1d84-4688-a794-b78b94668466 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:45:19.523117 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 636/1260] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Aug 28 00:45:19 2022] GET /identity/v3/role_assignments?user.id=c2a3834fa6894dada3291e4781bbd553&scope.project.id=f53e6fdb369b480fbebb4086a505df67&role.id=f08d6043261747e3b4763c744eef7dfb => generated 259 bytes in 237 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:45:21.558419 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be850f3f-aa8c-4297-a104-74b59c900f60 None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:45:21.559041 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be850f3f-aa8c-4297-a104-74b59c900f60 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:45:21.559041 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be850f3f-aa8c-4297-a104-74b59c900f60 None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:45:21.560666 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 625/1261] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:45:21 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:45:21.567329 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80308a2c-e52e-455b-8de6-9b0003341e58 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:45:21.567818 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80308a2c-e52e-455b-8de6-9b0003341e58 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:45:21.568179 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80308a2c-e52e-455b-8de6-9b0003341e58 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:45:21.597995 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-80308a2c-e52e-455b-8de6-9b0003341e58 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:45:21.599773 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-80308a2c-e52e-455b-8de6-9b0003341e58 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:45:21.686695 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 637/1262] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:45:21 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 28 00:45:21.956792 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be850f3f-aa8c-4297-a104-74b59c900f60 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:45:21.969172 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:45:22.070273 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be850f3f-aa8c-4297-a104-74b59c900f60 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:45:22.107236 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be34d1f2-147d-48d1-b6ee-3f8a342c094b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:45:22.107865 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be34d1f2-147d-48d1-b6ee-3f8a342c094b None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:45:22.108137 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be34d1f2-147d-48d1-b6ee-3f8a342c094b None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:45:22.108404 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be34d1f2-147d-48d1-b6ee-3f8a342c094b None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:45:22.120250 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be34d1f2-147d-48d1-b6ee-3f8a342c094b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 28 00:45:22.125166 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-be34d1f2-147d-48d1-b6ee-3f8a342c094b None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 28 00:45:22.125166 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:45:22.125166 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:45:22.125166 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:45:22.125166 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:45:22.125166 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:45:22.125166 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:45:22.125166 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:45:22.125166 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:45:22.125166 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:45:22.125166 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:45:22.125166 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:45:22.125166 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 28 00:45:22.125166 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 28 00:45:22.125166 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 28 00:45:22.125166 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 28 00:45:22.126863 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 28 00:45:22.126863 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 28 00:45:22.126863 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 28 00:45:22.126863 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 28 00:45:22.126863 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:45:22.126863 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:45:22.126863 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:45:22.126863 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:45:22.126863 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:45:22.126863 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:45:22.126863 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:45:22.126863 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:45:22.126863 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:45:22.126863 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:45:22.126863 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:45:22.126863 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:45:22.128446 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:45:22.128446 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:45:22.128446 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:45:22.128446 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:45:22.128446 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 28 00:45:22.128446 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 28 00:45:22.128446 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 28 00:45:22.128446 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 28 00:45:22.128446 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 28 00:45:22.128446 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 28 00:45:22.128446 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 28 00:45:22.128446 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:45:22.128446 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 626/1263] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Aug 28 00:45:21 2022] GET /identity/v3/roles/admin => generated 83 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:45:22.132498 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80308a2c-e52e-455b-8de6-9b0003341e58 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:45:22.152229 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:45:22.240241 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80308a2c-e52e-455b-8de6-9b0003341e58 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:45:22.275755 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27b1b70b-22ef-44cd-b52a-92fefd6459ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:45:22.276360 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27b1b70b-22ef-44cd-b52a-92fefd6459ea None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:45:22.276634 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27b1b70b-22ef-44cd-b52a-92fefd6459ea None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:45:22.276899 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27b1b70b-22ef-44cd-b52a-92fefd6459ea None admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:45:22.277589 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27b1b70b-22ef-44cd-b52a-92fefd6459ea None admin] RBAC: Adding query filter params (name=admin) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:45:22.277887 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27b1b70b-22ef-44cd-b52a-92fefd6459ea None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:45:22.279265 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27b1b70b-22ef-44cd-b52a-92fefd6459ea None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:45:22.288037 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 638/1264] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Aug 28 00:45:22 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:45:22.294185 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be34d1f2-147d-48d1-b6ee-3f8a342c094b None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:45:22.406846 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:45:22.463236 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be34d1f2-147d-48d1-b6ee-3f8a342c094b None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:45:22.520627 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7b1b4cc-443a-4d66-97fb-de98918d0061 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:45:22.521360 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7b1b4cc-443a-4d66-97fb-de98918d0061 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:45:22.521627 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7b1b4cc-443a-4d66-97fb-de98918d0061 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:45:22.521991 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7b1b4cc-443a-4d66-97fb-de98918d0061 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:45:22.537470 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7b1b4cc-443a-4d66-97fb-de98918d0061 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:45:22.540029 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7b1b4cc-443a-4d66-97fb-de98918d0061 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:45:22.561917 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-e7b1b4cc-443a-4d66-97fb-de98918d0061 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:45:22.561917 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:45:22.561917 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:45:22.561917 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:45:22.561917 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:45:22.561917 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:45:22.561917 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:45:22.561917 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:45:22.561917 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:45:22.561917 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:45:22.561917 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:45:22.561917 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:45:22.561917 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 28 00:45:22.561917 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 28 00:45:22.561917 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 28 00:45:22.561917 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 28 00:45:22.563471 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:45:22.563471 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:45:22.563471 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:45:22.563471 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:45:22.563471 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:45:22.563471 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:45:22.563471 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:45:22.563471 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:45:22.563471 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:45:22.563471 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:45:22.563471 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:45:22.563471 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:45:22.563471 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:45:22.563471 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:45:22.563471 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:45:22.563471 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:45:22.563471 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 28 00:45:22.564471 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 28 00:45:22.564471 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:45:22.564471 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:45:22.564471 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 627/1265] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:45:22 2022] GET /identity/v3/domains/Default => generated 87 bytes in 269 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:45:22.570288 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27b1b70b-22ef-44cd-b52a-92fefd6459ea None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:45:22.662088 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:45:22.683843 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27b1b70b-22ef-44cd-b52a-92fefd6459ea None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:45:22.719456 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd5e34a0-3d4d-4000-a031-b5fb3375170f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:45:22.720077 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd5e34a0-3d4d-4000-a031-b5fb3375170f None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:45:22.720355 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd5e34a0-3d4d-4000-a031-b5fb3375170f None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:45:22.720653 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd5e34a0-3d4d-4000-a031-b5fb3375170f None admin] PATH_INFO: `/v3/domains` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:45:22.721356 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd5e34a0-3d4d-4000-a031-b5fb3375170f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:45:22.721662 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd5e34a0-3d4d-4000-a031-b5fb3375170f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:45:22.723074 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd5e34a0-3d4d-4000-a031-b5fb3375170f None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:45:22.736226 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 639/1266] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Aug 28 00:45:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:45:22.742299 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7b1b4cc-443a-4d66-97fb-de98918d0061 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:45:22.848340 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7b1b4cc-443a-4d66-97fb-de98918d0061 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:45:22.869525 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:45:22.884360 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6db32bc6-ff34-42de-b671-b4b08e5cf176 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:45:22.885036 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6db32bc6-ff34-42de-b671-b4b08e5cf176 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:45:22.885321 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6db32bc6-ff34-42de-b671-b4b08e5cf176 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:45:22.885590 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6db32bc6-ff34-42de-b671-b4b08e5cf176 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:45:22.894626 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6db32bc6-ff34-42de-b671-b4b08e5cf176 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:45:22.896128 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6db32bc6-ff34-42de-b671-b4b08e5cf176 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:45:22.909229 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-6db32bc6-ff34-42de-b671-b4b08e5cf176 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Aug 28 00:45:22.909229 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:45:22.909229 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:45:22.909229 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:45:22.909229 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:45:22.909229 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:45:22.909229 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:45:22.909229 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:45:22.909229 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:45:22.909229 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:45:22.909229 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:45:22.909229 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:45:22.909229 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 28 00:45:22.909229 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 28 00:45:22.909229 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 28 00:45:22.909229 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 28 00:45:22.910432 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:45:22.910432 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:45:22.910432 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 28 00:45:22.910432 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 28 00:45:22.910432 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 28 00:45:22.910432 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 28 00:45:22.910432 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:45:22.910432 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:45:22.910432 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:45:22.910432 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:45:22.910432 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:45:22.910432 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:45:22.910432 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:45:22.910432 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:45:22.910432 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:45:22.910432 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:45:22.910432 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:45:22.910432 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:45:22.911562 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:45:22.911562 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:45:22.911562 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 28 00:45:22.911562 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 28 00:45:22.911562 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 28 00:45:22.911562 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 28 00:45:22.911562 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 28 00:45:22.911562 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 28 00:45:22.911562 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Aug 28 00:45:22.911562 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:45:22.911562 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 628/1267] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:45:22 2022] GET /identity/v3/users/placement => generated 87 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:45:22.919518 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd5e34a0-3d4d-4000-a031-b5fb3375170f None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:45:23.031506 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd5e34a0-3d4d-4000-a031-b5fb3375170f None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:45:23.071778 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7132abbc-90cc-4201-905f-cb84625d5cd2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:45:23.072411 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7132abbc-90cc-4201-905f-cb84625d5cd2 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:45:23.072702 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7132abbc-90cc-4201-905f-cb84625d5cd2 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:45:23.072986 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7132abbc-90cc-4201-905f-cb84625d5cd2 None admin] PATH_INFO: `/v3/users` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:45:23.073759 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7132abbc-90cc-4201-905f-cb84625d5cd2 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:45:23.074064 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7132abbc-90cc-4201-905f-cb84625d5cd2 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:45:23.075573 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7132abbc-90cc-4201-905f-cb84625d5cd2 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:45:23.086615 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:45:23.104310 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 640/1268] 10.222.0.42 () {56 vars in 1143 bytes} [Sun Aug 28 00:45:22 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:45:23.110614 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6db32bc6-ff34-42de-b671-b4b08e5cf176 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:45:23.232211 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6db32bc6-ff34-42de-b671-b4b08e5cf176 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:45:23.272920 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aa05a45-8287-4dc3-b84f-23e93a74cc1e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:45:23.273659 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aa05a45-8287-4dc3-b84f-23e93a74cc1e None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:45:23.273977 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aa05a45-8287-4dc3-b84f-23e93a74cc1e None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:45:23.274286 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aa05a45-8287-4dc3-b84f-23e93a74cc1e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:45:23.286618 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aa05a45-8287-4dc3-b84f-23e93a74cc1e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:45:23.288101 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aa05a45-8287-4dc3-b84f-23e93a74cc1e None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:45:23.304740 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-6aa05a45-8287-4dc3-b84f-23e93a74cc1e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:45:23.304740 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:45:23.304740 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:45:23.304740 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:45:23.304740 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:45:23.304740 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:45:23.304740 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:45:23.304740 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:45:23.304740 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:45:23.304740 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:45:23.304740 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:45:23.304740 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:45:23.304740 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 28 00:45:23.304740 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 28 00:45:23.304740 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 28 00:45:23.304740 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 28 00:45:23.306564 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:45:23.306564 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:45:23.306564 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:45:23.306564 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:45:23.306564 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:45:23.306564 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:45:23.306564 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:45:23.306564 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:45:23.306564 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:45:23.306564 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:45:23.306564 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:45:23.306564 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:45:23.306564 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:45:23.306564 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:45:23.306564 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:45:23.306564 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:45:23.306564 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 28 00:45:23.308334 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 28 00:45:23.308334 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:45:23.308334 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:45:23.308334 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 629/1269] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:45:23 2022] GET /identity/v3/domains/Default => generated 87 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:45:23.313436 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7132abbc-90cc-4201-905f-cb84625d5cd2 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:45:23.440851 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7132abbc-90cc-4201-905f-cb84625d5cd2 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:45:23.501512 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7de9d7f7-06b4-4ae2-85c5-4d338b08d8d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:45:23.502528 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7de9d7f7-06b4-4ae2-85c5-4d338b08d8d4 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:45:23.503016 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7de9d7f7-06b4-4ae2-85c5-4d338b08d8d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:45:23.503484 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7de9d7f7-06b4-4ae2-85c5-4d338b08d8d4 None admin] PATH_INFO: `/v3/domains` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:45:23.504640 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7de9d7f7-06b4-4ae2-85c5-4d338b08d8d4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:45:23.505204 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7de9d7f7-06b4-4ae2-85c5-4d338b08d8d4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:45:23.507887 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7de9d7f7-06b4-4ae2-85c5-4d338b08d8d4 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:45:23.524922 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 641/1270] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Aug 28 00:45:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:45:23.534747 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aa05a45-8287-4dc3-b84f-23e93a74cc1e None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:45:23.592556 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:45:23.715647 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aa05a45-8287-4dc3-b84f-23e93a74cc1e None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:45:23.775926 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c382d424-47bf-4126-b568-876df396caeb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:45:23.777006 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c382d424-47bf-4126-b568-876df396caeb None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:45:23.777516 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c382d424-47bf-4126-b568-876df396caeb None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:45:23.778006 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c382d424-47bf-4126-b568-876df396caeb None admin] PATH_INFO: `/v3/projects/service` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:45:23.789896 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c382d424-47bf-4126-b568-876df396caeb None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:45:23.792598 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c382d424-47bf-4126-b568-876df396caeb None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:45:23.813445 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-c382d424-47bf-4126-b568-876df396caeb None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 28 00:45:23.813445 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:45:23.813445 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:45:23.813445 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:45:23.813445 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:45:23.813445 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:45:23.813445 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:45:23.813445 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:45:23.813445 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:45:23.813445 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:45:23.813445 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:45:23.813445 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:45:23.813445 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 28 00:45:23.813445 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 28 00:45:23.813445 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 28 00:45:23.814531 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 28 00:45:23.814531 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:45:23.814531 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:45:23.814531 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:45:23.814531 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:45:23.814531 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:45:23.814531 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:45:23.814531 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:45:23.814531 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:45:23.814531 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:45:23.814531 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:45:23.814531 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:45:23.814531 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:45:23.814531 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:45:23.814531 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:45:23.814531 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:45:23.814531 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:45:23.815676 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 28 00:45:23.815676 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 28 00:45:23.815676 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 28 00:45:23.815676 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 28 00:45:23.815676 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 28 00:45:23.815676 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 28 00:45:23.815676 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 28 00:45:23.815676 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:45:23.815676 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 630/1271] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Aug 28 00:45:23 2022] GET /identity/v3/projects/service => generated 88 bytes in 282 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:45:23.821619 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7de9d7f7-06b4-4ae2-85c5-4d338b08d8d4 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:45:23.899889 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:45:23.944211 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7de9d7f7-06b4-4ae2-85c5-4d338b08d8d4 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:45:23.979732 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9fae610-b66b-421c-9d1b-cebcf6fd1181 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:45:23.980245 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9fae610-b66b-421c-9d1b-cebcf6fd1181 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:45:23.980519 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9fae610-b66b-421c-9d1b-cebcf6fd1181 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:45:23.980783 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9fae610-b66b-421c-9d1b-cebcf6fd1181 None admin] PATH_INFO: `/v3/projects` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:45:23.981506 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9fae610-b66b-421c-9d1b-cebcf6fd1181 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:45:23.981802 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9fae610-b66b-421c-9d1b-cebcf6fd1181 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:45:23.983241 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9fae610-b66b-421c-9d1b-cebcf6fd1181 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:45:23.996198 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 642/1272] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Aug 28 00:45:23 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:45:24.002626 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c382d424-47bf-4126-b568-876df396caeb None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:45:24.098751 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:45:24.108307 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c382d424-47bf-4126-b568-876df396caeb None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:45:24.144379 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01bf948f-680f-4af2-b056-f3cf71120f75 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:45:24.145070 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01bf948f-680f-4af2-b056-f3cf71120f75 None admin] REQUEST_METHOD: `PUT` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:45:24.145349 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01bf948f-680f-4af2-b056-f3cf71120f75 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:45:24.145624 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01bf948f-680f-4af2-b056-f3cf71120f75 None admin] PATH_INFO: `/v3/projects/f53e6fdb369b480fbebb4086a505df67/users/c2a3834fa6894dada3291e4781bbd553/roles/f08d6043261747e3b4763c744eef7dfb` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:45:24.147694 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01bf948f-680f-4af2-b056-f3cf71120f75 None admin] RBAC: Authorizing `identity:create_grant(project_id=f53e6fdb369b480fbebb4086a505df67, user_id=c2a3834fa6894dada3291e4781bbd553, role_id=f08d6043261747e3b4763c744eef7dfb)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:45:24.149237 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01bf948f-680f-4af2-b056-f3cf71120f75 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:45:24.164238 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 631/1273] 10.222.0.42 () {58 vars in 1439 bytes} [Sun Aug 28 00:45:24 2022] PUT /identity/v3/projects/f53e6fdb369b480fbebb4086a505df67/users/c2a3834fa6894dada3291e4781bbd553/roles/f08d6043261747e3b4763c744eef7dfb => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:45:26.081877 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f498bca-7b14-423b-894a-d27358827d73 None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:45:26.082534 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f498bca-7b14-423b-894a-d27358827d73 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:45:26.082534 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f498bca-7b14-423b-894a-d27358827d73 None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:45:26.083751 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 643/1274] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:45:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 28 00:45:26.089709 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a43e5f-43a1-4992-bdd7-d927bc6c9503 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:45:26.090538 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a43e5f-43a1-4992-bdd7-d927bc6c9503 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:45:26.090700 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a43e5f-43a1-4992-bdd7-d927bc6c9503 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:45:26.110212 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-59a43e5f-43a1-4992-bdd7-d927bc6c9503 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:45:26.111479 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-59a43e5f-43a1-4992-bdd7-d927bc6c9503 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:45:26.188874 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 632/1275] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:45:26 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 28 00:45:26.489375 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f498bca-7b14-423b-894a-d27358827d73 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:45:26.614743 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f498bca-7b14-423b-894a-d27358827d73 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:45:26.636700 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:45:26.655194 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-696be9e4-3378-411a-96d5-a3590141e78f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:45:26.655840 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-696be9e4-3378-411a-96d5-a3590141e78f None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:45:26.656117 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-696be9e4-3378-411a-96d5-a3590141e78f None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:45:26.656386 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-696be9e4-3378-411a-96d5-a3590141e78f None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:45:26.667551 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-696be9e4-3378-411a-96d5-a3590141e78f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Aug 28 00:45:26.672456 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-696be9e4-3378-411a-96d5-a3590141e78f None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Aug 28 00:45:26.672456 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:45:26.672456 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:45:26.672456 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:45:26.672456 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:45:26.672456 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:45:26.672456 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:45:26.672456 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:45:26.672456 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:45:26.672456 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:45:26.672456 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:45:26.672456 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:45:26.672456 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Aug 28 00:45:26.672456 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_role(role_id) Aug 28 00:45:26.672456 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Aug 28 00:45:26.672456 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Aug 28 00:45:26.674247 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Aug 28 00:45:26.674247 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Aug 28 00:45:26.674247 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Aug 28 00:45:26.674247 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Aug 28 00:45:26.674247 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:45:26.674247 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:45:26.674247 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:45:26.674247 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:45:26.674247 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:45:26.674247 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:45:26.674247 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:45:26.674247 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:45:26.674247 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:45:26.674247 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:45:26.674247 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:45:26.674247 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:45:26.675912 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:45:26.675912 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:45:26.675912 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:45:26.675912 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:45:26.675912 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Aug 28 00:45:26.675912 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Aug 28 00:45:26.675912 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Aug 28 00:45:26.675912 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Aug 28 00:45:26.675912 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Aug 28 00:45:26.675912 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Aug 28 00:45:26.675912 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Aug 28 00:45:26.675912 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:45:26.675912 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 644/1276] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Aug 28 00:45:26 2022] GET /identity/v3/roles/admin => generated 83 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:45:26.680327 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59a43e5f-43a1-4992-bdd7-d927bc6c9503 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:45:26.768848 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:45:26.803106 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59a43e5f-43a1-4992-bdd7-d927bc6c9503 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:45:26.863596 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4741c741-f965-4006-8ff2-e2dfd07e79cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:45:26.864622 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4741c741-f965-4006-8ff2-e2dfd07e79cb None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:45:26.865137 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4741c741-f965-4006-8ff2-e2dfd07e79cb None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:45:26.865595 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4741c741-f965-4006-8ff2-e2dfd07e79cb None admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:45:26.866830 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4741c741-f965-4006-8ff2-e2dfd07e79cb None admin] RBAC: Adding query filter params (name=admin) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:45:26.867343 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4741c741-f965-4006-8ff2-e2dfd07e79cb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:45:26.869874 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4741c741-f965-4006-8ff2-e2dfd07e79cb None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:45:26.885569 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 633/1277] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Aug 28 00:45:26 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 28 00:45:26.891756 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-696be9e4-3378-411a-96d5-a3590141e78f None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:45:26.998513 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-696be9e4-3378-411a-96d5-a3590141e78f None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:45:27.042241 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b42c399a-29eb-4ec0-81bb-eee05e002bca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:45:27.042787 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b42c399a-29eb-4ec0-81bb-eee05e002bca None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:45:27.043060 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b42c399a-29eb-4ec0-81bb-eee05e002bca None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:45:27.043326 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b42c399a-29eb-4ec0-81bb-eee05e002bca None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:45:27.054971 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b42c399a-29eb-4ec0-81bb-eee05e002bca None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:45:27.056440 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b42c399a-29eb-4ec0-81bb-eee05e002bca None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:45:27.071770 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-b42c399a-29eb-4ec0-81bb-eee05e002bca None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:45:27.071770 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:45:27.071770 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:45:27.071770 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:45:27.071770 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:45:27.071770 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:45:27.071770 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:45:27.071770 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:45:27.071770 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:45:27.071770 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:45:27.071770 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:45:27.071770 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:45:27.071770 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 28 00:45:27.071770 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 28 00:45:27.071770 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 28 00:45:27.071770 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 28 00:45:27.073634 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:45:27.073634 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:45:27.073634 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:45:27.073634 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:45:27.073634 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:45:27.073634 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:45:27.073634 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:45:27.073634 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:45:27.073634 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:45:27.073634 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:45:27.073634 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:45:27.073634 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:45:27.073634 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:45:27.073634 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:45:27.073634 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:45:27.073634 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:45:27.073634 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 28 00:45:27.075788 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 28 00:45:27.075788 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:45:27.075788 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:45:27.075788 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 645/1278] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:45:26 2022] GET /identity/v3/domains/Default => generated 87 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:45:27.080710 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4741c741-f965-4006-8ff2-e2dfd07e79cb None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:45:27.192851 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4741c741-f965-4006-8ff2-e2dfd07e79cb None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:45:27.229077 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78dda663-9880-4b42-8f6e-a37443801177 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:45:27.229765 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78dda663-9880-4b42-8f6e-a37443801177 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:45:27.230072 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78dda663-9880-4b42-8f6e-a37443801177 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:45:27.230449 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78dda663-9880-4b42-8f6e-a37443801177 None admin] PATH_INFO: `/v3/domains` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:45:27.231184 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78dda663-9880-4b42-8f6e-a37443801177 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:45:27.231573 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78dda663-9880-4b42-8f6e-a37443801177 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:45:27.233063 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78dda663-9880-4b42-8f6e-a37443801177 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:45:27.246440 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 634/1279] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Aug 28 00:45:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 28 00:45:27.252367 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b42c399a-29eb-4ec0-81bb-eee05e002bca None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:45:27.286470 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:45:27.371405 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b42c399a-29eb-4ec0-81bb-eee05e002bca None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:45:27.411397 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2b2de3a-0b9c-465d-aaac-f0916779adc8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:45:27.412054 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2b2de3a-0b9c-465d-aaac-f0916779adc8 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:45:27.412335 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2b2de3a-0b9c-465d-aaac-f0916779adc8 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:45:27.412609 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2b2de3a-0b9c-465d-aaac-f0916779adc8 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:45:27.421939 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2b2de3a-0b9c-465d-aaac-f0916779adc8 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:45:27.423571 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2b2de3a-0b9c-465d-aaac-f0916779adc8 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:45:27.436920 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-f2b2de3a-0b9c-465d-aaac-f0916779adc8 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Aug 28 00:45:27.436920 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:45:27.436920 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:45:27.436920 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:45:27.436920 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:45:27.436920 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:45:27.436920 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:45:27.436920 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:45:27.436920 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:45:27.436920 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:45:27.436920 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:45:27.436920 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:45:27.436920 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Aug 28 00:45:27.436920 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_user(user_id) Aug 28 00:45:27.436920 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Aug 28 00:45:27.436920 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Aug 28 00:45:27.438810 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:45:27.438810 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:45:27.438810 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Aug 28 00:45:27.438810 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 28 00:45:27.438810 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Aug 28 00:45:27.438810 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Aug 28 00:45:27.438810 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:45:27.438810 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:45:27.438810 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:45:27.438810 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:45:27.438810 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:45:27.438810 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:45:27.438810 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:45:27.438810 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:45:27.438810 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:45:27.438810 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:45:27.438810 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:45:27.438810 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:45:27.441038 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:45:27.441038 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:45:27.441038 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Aug 28 00:45:27.441038 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Aug 28 00:45:27.441038 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Aug 28 00:45:27.441038 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Aug 28 00:45:27.441038 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Aug 28 00:45:27.441038 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Aug 28 00:45:27.441038 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Aug 28 00:45:27.441038 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:45:27.441038 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 646/1280] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:45:27 2022] GET /identity/v3/users/placement => generated 87 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:45:27.446174 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78dda663-9880-4b42-8f6e-a37443801177 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:45:27.450513 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:45:27.570733 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78dda663-9880-4b42-8f6e-a37443801177 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:45:27.608894 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e18f920-10bc-4b98-b64b-7aa383fe8766 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:45:27.609602 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e18f920-10bc-4b98-b64b-7aa383fe8766 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:45:27.609923 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e18f920-10bc-4b98-b64b-7aa383fe8766 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:45:27.610236 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e18f920-10bc-4b98-b64b-7aa383fe8766 None admin] PATH_INFO: `/v3/users` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:45:27.611009 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e18f920-10bc-4b98-b64b-7aa383fe8766 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:45:27.611354 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e18f920-10bc-4b98-b64b-7aa383fe8766 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:45:27.612831 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e18f920-10bc-4b98-b64b-7aa383fe8766 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:45:27.632812 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 635/1281] 10.222.0.42 () {56 vars in 1143 bytes} [Sun Aug 28 00:45:27 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 28 00:45:27.639335 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2b2de3a-0b9c-465d-aaac-f0916779adc8 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:45:27.743520 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:45:27.755007 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2b2de3a-0b9c-465d-aaac-f0916779adc8 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:45:27.790346 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22372f00-0306-4bb4-b64b-1419e2440b8e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:45:27.790957 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22372f00-0306-4bb4-b64b-1419e2440b8e None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:45:27.791228 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22372f00-0306-4bb4-b64b-1419e2440b8e None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:45:27.791494 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22372f00-0306-4bb4-b64b-1419e2440b8e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:45:27.863770 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22372f00-0306-4bb4-b64b-1419e2440b8e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:45:27.865250 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22372f00-0306-4bb4-b64b-1419e2440b8e None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:45:27.959081 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:45:27.970997 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 647/1282] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Aug 28 00:45:27 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 332 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Aug 28 00:45:27.980803 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e18f920-10bc-4b98-b64b-7aa383fe8766 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:45:28.047576 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:45:28.089161 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e18f920-10bc-4b98-b64b-7aa383fe8766 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:45:28.128236 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd675b3a-3476-4ed9-a450-33d1dcbfe651 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:45:28.128938 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd675b3a-3476-4ed9-a450-33d1dcbfe651 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:45:28.129300 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd675b3a-3476-4ed9-a450-33d1dcbfe651 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:45:28.129606 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd675b3a-3476-4ed9-a450-33d1dcbfe651 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:45:28.143011 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd675b3a-3476-4ed9-a450-33d1dcbfe651 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:45:28.144502 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd675b3a-3476-4ed9-a450-33d1dcbfe651 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:45:28.160544 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-cd675b3a-3476-4ed9-a450-33d1dcbfe651 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:45:28.160544 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:45:28.160544 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:45:28.160544 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:45:28.160544 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:45:28.160544 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:45:28.160544 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:45:28.160544 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:45:28.160544 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:45:28.160544 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:45:28.160544 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:45:28.160544 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:45:28.160544 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Aug 28 00:45:28.160544 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Aug 28 00:45:28.160544 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Aug 28 00:45:28.160544 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Aug 28 00:45:28.161821 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:45:28.161821 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:45:28.161821 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:45:28.161821 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:45:28.161821 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:45:28.161821 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:45:28.161821 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:45:28.161821 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:45:28.161821 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:45:28.161821 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:45:28.161821 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:45:28.161821 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:45:28.161821 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:45:28.161821 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:45:28.161821 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:45:28.161821 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:45:28.161821 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Aug 28 00:45:28.163135 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Aug 28 00:45:28.163135 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Aug 28 00:45:28.163135 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:45:28.163135 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 636/1283] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:45:27 2022] GET /identity/v3/domains/Default => generated 87 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:45:28.168125 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22372f00-0306-4bb4-b64b-1419e2440b8e None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:45:28.271469 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22372f00-0306-4bb4-b64b-1419e2440b8e None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:45:28.308338 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fe274f1-f46d-488f-8fef-eb083149b4c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:45:28.309156 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe274f1-f46d-488f-8fef-eb083149b4c3 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:45:28.309528 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe274f1-f46d-488f-8fef-eb083149b4c3 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:45:28.309894 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe274f1-f46d-488f-8fef-eb083149b4c3 None admin] PATH_INFO: `/v3/domains` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:45:28.310647 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fe274f1-f46d-488f-8fef-eb083149b4c3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:45:28.311046 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fe274f1-f46d-488f-8fef-eb083149b4c3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:45:28.312535 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fe274f1-f46d-488f-8fef-eb083149b4c3 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:45:28.325046 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 648/1284] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Aug 28 00:45:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:45:28.331461 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd675b3a-3476-4ed9-a450-33d1dcbfe651 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:45:28.439977 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd675b3a-3476-4ed9-a450-33d1dcbfe651 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:45:28.456077 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:45:28.479574 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af892415-315b-4c21-a59e-94e9782555b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:45:28.480271 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af892415-315b-4c21-a59e-94e9782555b4 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:45:28.480594 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af892415-315b-4c21-a59e-94e9782555b4 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:45:28.480908 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af892415-315b-4c21-a59e-94e9782555b4 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:45:28.488712 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af892415-315b-4c21-a59e-94e9782555b4 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:45:28.490307 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af892415-315b-4c21-a59e-94e9782555b4 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:45:28.504094 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application [None req-af892415-315b-4c21-a59e-94e9782555b4 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Aug 28 00:45:28.504094 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:45:28.504094 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:45:28.504094 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:45:28.504094 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:45:28.504094 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:45:28.504094 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:45:28.504094 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:45:28.504094 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:45:28.504094 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:45:28.504094 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:45:28.504094 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:45:28.504094 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Aug 28 00:45:28.504094 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_project(project_id) Aug 28 00:45:28.504094 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Aug 28 00:45:28.505883 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Aug 28 00:45:28.505883 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:45:28.505883 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:45:28.505883 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:45:28.505883 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:45:28.505883 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:45:28.505883 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:45:28.505883 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:45:28.505883 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application with Lock( Aug 28 00:45:28.505883 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:45:28.505883 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:45:28.505883 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:45:28.505883 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:45:28.505883 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:45:28.505883 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:45:28.505883 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:45:28.505883 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:45:28.507670 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Aug 28 00:45:28.507670 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Aug 28 00:45:28.507670 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Aug 28 00:45:28.507670 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Aug 28 00:45:28.507670 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Aug 28 00:45:28.507670 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Aug 28 00:45:28.507670 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Aug 28 00:45:28.507670 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: ERROR keystone.server.flask.application Aug 28 00:45:28.507670 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 637/1285] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Aug 28 00:45:28 2022] GET /identity/v3/projects/service => generated 88 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Aug 28 00:45:28.511872 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fe274f1-f46d-488f-8fef-eb083149b4c3 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:45:28.573018 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:45:28.641853 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fe274f1-f46d-488f-8fef-eb083149b4c3 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:45:28.681122 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0e43b39-bca9-4503-bccd-f3923f28f6f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:45:28.681734 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e43b39-bca9-4503-bccd-f3923f28f6f9 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:45:28.682009 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e43b39-bca9-4503-bccd-f3923f28f6f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:45:28.682278 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e43b39-bca9-4503-bccd-f3923f28f6f9 None admin] PATH_INFO: `/v3/projects` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:45:28.682974 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0e43b39-bca9-4503-bccd-f3923f28f6f9 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:45:28.683277 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0e43b39-bca9-4503-bccd-f3923f28f6f9 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:45:28.684729 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0e43b39-bca9-4503-bccd-f3923f28f6f9 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:45:28.697993 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 649/1286] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Aug 28 00:45:28 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:45:28.704903 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af892415-315b-4c21-a59e-94e9782555b4 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:45:28.813329 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af892415-315b-4c21-a59e-94e9782555b4 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:45:28.850295 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b160041-622b-4e75-9d59-b571a7eb3c3a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:45:28.850860 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b160041-622b-4e75-9d59-b571a7eb3c3a None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:45:28.851196 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b160041-622b-4e75-9d59-b571a7eb3c3a None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:45:28.851466 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b160041-622b-4e75-9d59-b571a7eb3c3a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:45:28.852196 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b160041-622b-4e75-9d59-b571a7eb3c3a None admin] RBAC: Adding query filter params (role.id=f08d6043261747e3b4763c744eef7dfb, scope.project.id=f53e6fdb369b480fbebb4086a505df67, user.id=c2a3834fa6894dada3291e4781bbd553) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:45:28.852499 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b160041-622b-4e75-9d59-b571a7eb3c3a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:45:28.854070 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b160041-622b-4e75-9d59-b571a7eb3c3a None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:45:28.860593 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 638/1287] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Aug 28 00:45:28 2022] GET /identity/v3/role_assignments?user.id=c2a3834fa6894dada3291e4781bbd553&scope.project.id=f53e6fdb369b480fbebb4086a505df67&role.id=f08d6043261747e3b4763c744eef7dfb => generated 608 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 28 00:45:30.797107 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c3b26e7-78d1-4161-8c05-5dc2ae9e7acb None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:45:30.797954 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c3b26e7-78d1-4161-8c05-5dc2ae9e7acb None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:45:30.797954 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c3b26e7-78d1-4161-8c05-5dc2ae9e7acb None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:45:30.798997 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 650/1288] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:45:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:45:30.804781 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-450e1556-b257-4fcf-a2eb-0966fc6fe40b None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:45:30.805351 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-450e1556-b257-4fcf-a2eb-0966fc6fe40b None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:45:30.805351 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-450e1556-b257-4fcf-a2eb-0966fc6fe40b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:45:30.824240 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-450e1556-b257-4fcf-a2eb-0966fc6fe40b None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:45:30.825468 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-450e1556-b257-4fcf-a2eb-0966fc6fe40b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:45:30.859345 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:45:30.898610 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 639/1289] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:45:30 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 28 00:45:31.170096 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c3b26e7-78d1-4161-8c05-5dc2ae9e7acb None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:45:31.276615 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c3b26e7-78d1-4161-8c05-5dc2ae9e7acb None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:45:31.288363 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:45:31.314581 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ebda6be-5dab-489b-b688-0ab27cb8acb5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:45:31.315212 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ebda6be-5dab-489b-b688-0ab27cb8acb5 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:45:31.315488 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ebda6be-5dab-489b-b688-0ab27cb8acb5 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:45:31.315759 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ebda6be-5dab-489b-b688-0ab27cb8acb5 None admin] PATH_INFO: `/v3/services/placement` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:45:31.316340 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ebda6be-5dab-489b-b688-0ab27cb8acb5 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:45:31.317780 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ebda6be-5dab-489b-b688-0ab27cb8acb5 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:45:31.327718 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-2ebda6be-5dab-489b-b688-0ab27cb8acb5 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Aug 28 00:45:31.327718 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:45:31.327718 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 28 00:45:31.327718 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 28 00:45:31.327718 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 28 00:45:31.327718 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 28 00:45:31.327718 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 28 00:45:31.327718 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 28 00:45:31.327718 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Aug 28 00:45:31.327718 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:45:31.327718 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 28 00:45:31.327718 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:45:31.327718 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:45:31.327718 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:45:31.327718 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:45:31.327718 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:45:31.327718 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:45:31.328954 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:45:31.328954 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:45:31.328954 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:45:31.328954 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:45:31.328954 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:45:31.328954 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:45:31.328954 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 28 00:45:31.328954 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 28 00:45:31.328954 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 28 00:45:31.328954 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 28 00:45:31.328954 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:45:31.328954 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:45:31.328954 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:45:31.328954 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:45:31.328954 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:45:31.328954 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:45:31.330004 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:45:31.330004 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:45:31.330004 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:45:31.330004 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:45:31.330004 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:45:31.330004 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:45:31.330004 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:45:31.330004 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:45:31.330004 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:45:31.330004 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:45:31.330004 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 28 00:45:31.330004 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 28 00:45:31.330004 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Aug 28 00:45:31.330004 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:45:31.330004 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 651/1290] 10.222.0.42 () {56 vars in 1117 bytes} [Sun Aug 28 00:45:31 2022] GET /identity/v3/services/placement => generated 90 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:45:31.338270 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-450e1556-b257-4fcf-a2eb-0966fc6fe40b None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:45:31.448060 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-450e1556-b257-4fcf-a2eb-0966fc6fe40b None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:45:31.484437 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de267c50-49ed-408b-93fe-71ee2a37a705 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:45:31.485085 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de267c50-49ed-408b-93fe-71ee2a37a705 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:45:31.485361 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de267c50-49ed-408b-93fe-71ee2a37a705 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:45:31.485628 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de267c50-49ed-408b-93fe-71ee2a37a705 None admin] PATH_INFO: `/v3/services` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:45:31.486247 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de267c50-49ed-408b-93fe-71ee2a37a705 None admin] RBAC: Adding query filter params (name=placement) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:45:31.486543 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de267c50-49ed-408b-93fe-71ee2a37a705 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:45:31.487918 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de267c50-49ed-408b-93fe-71ee2a37a705 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:45:31.493200 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 640/1291] 10.222.0.42 () {56 vars in 1116 bytes} [Sun Aug 28 00:45:31 2022] GET /identity/v3/services?name=placement => generated 126 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:45:31.499171 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ebda6be-5dab-489b-b688-0ab27cb8acb5 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:45:31.635009 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ebda6be-5dab-489b-b688-0ab27cb8acb5 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:45:31.671188 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5669f1eb-983d-41e1-9255-e71c65f5693e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:45:31.671798 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5669f1eb-983d-41e1-9255-e71c65f5693e None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:45:31.672074 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5669f1eb-983d-41e1-9255-e71c65f5693e None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:45:31.672342 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5669f1eb-983d-41e1-9255-e71c65f5693e None admin] PATH_INFO: `/v3/services` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:45:31.672998 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5669f1eb-983d-41e1-9255-e71c65f5693e None admin] RBAC: Adding query filter params (type=placement) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:45:31.673297 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5669f1eb-983d-41e1-9255-e71c65f5693e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:45:31.674686 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5669f1eb-983d-41e1-9255-e71c65f5693e None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:45:31.679766 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 652/1292] 10.222.0.42 () {56 vars in 1116 bytes} [Sun Aug 28 00:45:31 2022] GET /identity/v3/services?type=placement => generated 126 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:45:33.603372 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a688153-2079-4884-a4a4-c0f8528185f2 None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:45:33.603965 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a688153-2079-4884-a4a4-c0f8528185f2 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:45:33.603965 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a688153-2079-4884-a4a4-c0f8528185f2 None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:45:33.605292 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 641/1293] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:45:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:45:33.611117 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-181c2ba7-ec38-4f18-b4cd-a35097b67fc4 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:45:33.611669 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-181c2ba7-ec38-4f18-b4cd-a35097b67fc4 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:45:33.611669 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-181c2ba7-ec38-4f18-b4cd-a35097b67fc4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:45:33.631074 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-181c2ba7-ec38-4f18-b4cd-a35097b67fc4 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:45:33.632299 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-181c2ba7-ec38-4f18-b4cd-a35097b67fc4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:45:33.671379 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:45:33.738626 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 653/1294] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:45:33 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 28 00:45:34.239689 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a688153-2079-4884-a4a4-c0f8528185f2 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:45:34.276212 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:45:34.353520 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a688153-2079-4884-a4a4-c0f8528185f2 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:45:34.390963 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f81b29f-33c4-46fa-b6a7-cfdf99f35a91 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:45:34.391584 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f81b29f-33c4-46fa-b6a7-cfdf99f35a91 None admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:45:34.391860 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f81b29f-33c4-46fa-b6a7-cfdf99f35a91 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:45:34.392123 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f81b29f-33c4-46fa-b6a7-cfdf99f35a91 None admin] PATH_INFO: `/v3/services` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:45:34.392832 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f81b29f-33c4-46fa-b6a7-cfdf99f35a91 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:45:34.394342 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f81b29f-33c4-46fa-b6a7-cfdf99f35a91 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:45:34.411893 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 642/1295] 10.222.0.42 () {60 vars in 1141 bytes} [Sun Aug 28 00:45:34 2022] POST /identity/v3/services => generated 246 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:45:36.396922 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cac26d6e-49f3-4caa-b4e9-215e4c270131 None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:45:36.397608 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cac26d6e-49f3-4caa-b4e9-215e4c270131 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:45:36.397608 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cac26d6e-49f3-4caa-b4e9-215e4c270131 None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:45:36.399017 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 654/1296] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:45:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:45:36.405252 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6e46d43-cb2f-42f8-bf32-2acd6f17fc15 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:45:36.406103 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6e46d43-cb2f-42f8-bf32-2acd6f17fc15 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:45:36.406262 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6e46d43-cb2f-42f8-bf32-2acd6f17fc15 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:45:36.425695 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-e6e46d43-cb2f-42f8-bf32-2acd6f17fc15 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:45:36.426962 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-e6e46d43-cb2f-42f8-bf32-2acd6f17fc15 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:45:36.506222 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 643/1297] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:45:36 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 28 00:45:36.789999 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cac26d6e-49f3-4caa-b4e9-215e4c270131 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:45:36.900636 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cac26d6e-49f3-4caa-b4e9-215e4c270131 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:45:36.942147 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8fab023-9840-4ce2-bbfd-c25a9394f6d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:45:36.942879 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8fab023-9840-4ce2-bbfd-c25a9394f6d0 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:45:36.943201 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8fab023-9840-4ce2-bbfd-c25a9394f6d0 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:45:36.943514 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8fab023-9840-4ce2-bbfd-c25a9394f6d0 None admin] PATH_INFO: `/v3/services/placement` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:45:36.944191 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8fab023-9840-4ce2-bbfd-c25a9394f6d0 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:45:36.945770 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8fab023-9840-4ce2-bbfd-c25a9394f6d0 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:45:36.956278 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-a8fab023-9840-4ce2-bbfd-c25a9394f6d0 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Aug 28 00:45:36.956278 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:45:36.956278 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 28 00:45:36.956278 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 28 00:45:36.956278 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 28 00:45:36.956278 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 28 00:45:36.956278 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 28 00:45:36.956278 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 28 00:45:36.956278 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Aug 28 00:45:36.956278 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:45:36.956278 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 28 00:45:36.956278 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:45:36.956278 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:45:36.956278 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:45:36.956278 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:45:36.956278 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:45:36.956278 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:45:36.958228 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:45:36.958228 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:45:36.958228 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:45:36.958228 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:45:36.958228 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:45:36.958228 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:45:36.958228 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 28 00:45:36.958228 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 28 00:45:36.958228 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 28 00:45:36.958228 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 28 00:45:36.958228 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:45:36.958228 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:45:36.958228 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:45:36.958228 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:45:36.958228 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:45:36.958228 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:45:36.960082 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:45:36.960082 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:45:36.960082 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:45:36.960082 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:45:36.960082 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:45:36.960082 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:45:36.960082 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:45:36.960082 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:45:36.960082 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:45:36.960082 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:45:36.960082 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 28 00:45:36.960082 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 28 00:45:36.960082 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Aug 28 00:45:36.960082 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:45:36.960082 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 655/1298] 10.222.0.42 () {56 vars in 1117 bytes} [Sun Aug 28 00:45:36 2022] GET /identity/v3/services/placement => generated 90 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:45:36.965741 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6e46d43-cb2f-42f8-bf32-2acd6f17fc15 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:45:36.986341 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:45:37.074325 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6e46d43-cb2f-42f8-bf32-2acd6f17fc15 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:45:37.110456 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4300d2b-3831-4c59-baaa-47aba2ba0fe6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:45:37.111189 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4300d2b-3831-4c59-baaa-47aba2ba0fe6 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:45:37.111578 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4300d2b-3831-4c59-baaa-47aba2ba0fe6 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:45:37.111968 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4300d2b-3831-4c59-baaa-47aba2ba0fe6 None admin] PATH_INFO: `/v3/services` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:45:37.112781 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4300d2b-3831-4c59-baaa-47aba2ba0fe6 None admin] RBAC: Adding query filter params (name=placement) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:45:37.113231 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4300d2b-3831-4c59-baaa-47aba2ba0fe6 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:45:37.114814 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4300d2b-3831-4c59-baaa-47aba2ba0fe6 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:45:37.121016 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 644/1299] 10.222.0.42 () {56 vars in 1116 bytes} [Sun Aug 28 00:45:36 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:45:37.127252 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8fab023-9840-4ce2-bbfd-c25a9394f6d0 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:45:37.167256 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:45:37.240217 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8fab023-9840-4ce2-bbfd-c25a9394f6d0 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:45:37.275690 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5fa79ab-4f19-4a50-a97e-c7ce981a63a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:45:37.276300 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5fa79ab-4f19-4a50-a97e-c7ce981a63a6 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:45:37.276578 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5fa79ab-4f19-4a50-a97e-c7ce981a63a6 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:45:37.276837 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5fa79ab-4f19-4a50-a97e-c7ce981a63a6 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:45:37.277590 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5fa79ab-4f19-4a50-a97e-c7ce981a63a6 None admin] RBAC: Adding query filter params (interface=public, service_id=c054212deecd405ab3c996f2492828bc, region_id=RegionOne) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:45:37.277893 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5fa79ab-4f19-4a50-a97e-c7ce981a63a6 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:45:37.279276 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5fa79ab-4f19-4a50-a97e-c7ce981a63a6 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:45:37.284944 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 656/1300] 10.222.0.42 () {56 vars in 1251 bytes} [Sun Aug 28 00:45:37 2022] GET /identity/v3/endpoints?service_id=c054212deecd405ab3c996f2492828bc&interface=public®ion_id=RegionOne => generated 194 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:45:37.291564 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4300d2b-3831-4c59-baaa-47aba2ba0fe6 None admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:45:37.382385 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:45:37.404497 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4300d2b-3831-4c59-baaa-47aba2ba0fe6 None admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:45:37.440972 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17111fd6-26e2-4382-8d3f-9bd21fd9d437 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:45:37.441726 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17111fd6-26e2-4382-8d3f-9bd21fd9d437 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:45:37.441955 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17111fd6-26e2-4382-8d3f-9bd21fd9d437 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:45:37.442279 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17111fd6-26e2-4382-8d3f-9bd21fd9d437 None admin] PATH_INFO: `/v3/services` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:45:37.442940 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17111fd6-26e2-4382-8d3f-9bd21fd9d437 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:45:37.444417 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17111fd6-26e2-4382-8d3f-9bd21fd9d437 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:45:37.452317 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 645/1301] 10.222.0.42 () {56 vars in 1087 bytes} [Sun Aug 28 00:45:37 2022] GET /identity/v3/services => generated 2191 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 00:45:39.361030 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dbf9668-21af-4323-8cc0-9ae11272a432 None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:45:39.361629 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dbf9668-21af-4323-8cc0-9ae11272a432 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:45:39.361629 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dbf9668-21af-4323-8cc0-9ae11272a432 None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:45:39.362875 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 657/1302] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:45:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:45:39.368668 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6afbbf8d-7245-4ffe-ae5c-a311cf84742b None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:45:39.369138 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6afbbf8d-7245-4ffe-ae5c-a311cf84742b None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:45:39.369138 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6afbbf8d-7245-4ffe-ae5c-a311cf84742b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:45:39.388366 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-6afbbf8d-7245-4ffe-ae5c-a311cf84742b None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:45:39.389577 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-6afbbf8d-7245-4ffe-ae5c-a311cf84742b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:45:39.463372 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 646/1303] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:45:39 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Aug 28 00:45:39.727342 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dbf9668-21af-4323-8cc0-9ae11272a432 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:45:39.830209 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:45:39.838631 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dbf9668-21af-4323-8cc0-9ae11272a432 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:45:39.874863 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ce81dc9-7249-4043-ab20-e91ea7828739 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:45:39.875540 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce81dc9-7249-4043-ab20-e91ea7828739 None admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:45:39.875870 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce81dc9-7249-4043-ab20-e91ea7828739 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:45:39.876207 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce81dc9-7249-4043-ab20-e91ea7828739 None admin] PATH_INFO: `/v3/services/placement` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:45:39.876893 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ce81dc9-7249-4043-ab20-e91ea7828739 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:45:39.878397 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ce81dc9-7249-4043-ab20-e91ea7828739 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:45:39.887865 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application [None req-8ce81dc9-7249-4043-ab20-e91ea7828739 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Aug 28 00:45:39.887865 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:45:39.887865 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Aug 28 00:45:39.887865 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Aug 28 00:45:39.887865 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Aug 28 00:45:39.887865 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Aug 28 00:45:39.887865 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Aug 28 00:45:39.887865 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 28 00:45:39.887865 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Aug 28 00:45:39.887865 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:45:39.887865 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Aug 28 00:45:39.887865 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:45:39.887865 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Traceback (most recent call last): Aug 28 00:45:39.887865 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Aug 28 00:45:39.887865 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application rv = self.dispatch_request() Aug 28 00:45:39.887865 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Aug 28 00:45:39.887865 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Aug 28 00:45:39.889848 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Aug 28 00:45:39.889848 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Aug 28 00:45:39.889848 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Aug 28 00:45:39.889848 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Aug 28 00:45:39.889848 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Aug 28 00:45:39.889848 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Aug 28 00:45:39.889848 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Aug 28 00:45:39.889848 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._get_service(service_id) Aug 28 00:45:39.889848 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Aug 28 00:45:39.889848 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Aug 28 00:45:39.889848 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Aug 28 00:45:39.889848 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Aug 28 00:45:39.889848 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Aug 28 00:45:39.889848 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Aug 28 00:45:39.889848 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Aug 28 00:45:39.889848 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.get_or_create( Aug 28 00:45:39.891445 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Aug 28 00:45:39.891445 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application with Lock( Aug 28 00:45:39.891445 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 28 00:45:39.891445 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self._enter() Aug 28 00:45:39.891445 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Aug 28 00:45:39.891445 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Aug 28 00:45:39.891445 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 28 00:45:39.891445 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application return self.creator() Aug 28 00:45:39.891445 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Aug 28 00:45:39.891445 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application created_value = creator( Aug 28 00:45:39.891445 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Aug 28 00:45:39.891445 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Aug 28 00:45:39.891445 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Aug 28 00:45:39.891445 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: ERROR keystone.server.flask.application Aug 28 00:45:39.891445 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 658/1304] 10.222.0.42 () {56 vars in 1117 bytes} [Sun Aug 28 00:45:39 2022] GET /identity/v3/services/placement => generated 90 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Aug 28 00:45:39.896693 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6afbbf8d-7245-4ffe-ae5c-a311cf84742b None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:45:39.980749 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:45:40.006531 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6afbbf8d-7245-4ffe-ae5c-a311cf84742b None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:45:40.044256 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8a03b0d-c774-48cd-ae07-64134d0a9154 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:45:40.044873 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8a03b0d-c774-48cd-ae07-64134d0a9154 None admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:45:40.045176 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8a03b0d-c774-48cd-ae07-64134d0a9154 None admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:45:40.045444 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8a03b0d-c774-48cd-ae07-64134d0a9154 None admin] PATH_INFO: `/v3/services` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:45:40.046083 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8a03b0d-c774-48cd-ae07-64134d0a9154 None admin] RBAC: Adding query filter params (name=placement) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:45:40.046380 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8a03b0d-c774-48cd-ae07-64134d0a9154 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:45:40.047777 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8a03b0d-c774-48cd-ae07-64134d0a9154 None admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:45:40.053769 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 647/1305] 10.222.0.42 () {56 vars in 1116 bytes} [Sun Aug 28 00:45:39 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:45:40.059820 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ce81dc9-7249-4043-ab20-e91ea7828739 None admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:45:40.166398 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ce81dc9-7249-4043-ab20-e91ea7828739 None admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:45:40.201141 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a5f9f43-3db2-44ec-9d3a-cd4563d58f43 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:45:40.202055 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a5f9f43-3db2-44ec-9d3a-cd4563d58f43 None admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:45:40.202335 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a5f9f43-3db2-44ec-9d3a-cd4563d58f43 None admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:45:40.203600 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a5f9f43-3db2-44ec-9d3a-cd4563d58f43 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:45:40.203600 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a5f9f43-3db2-44ec-9d3a-cd4563d58f43 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:45:40.204627 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a5f9f43-3db2-44ec-9d3a-cd4563d58f43 None admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:45:40.226035 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 659/1306] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 00:45:40 2022] POST /identity/v3/endpoints => generated 331 bytes in 167 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:45:50.050537 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=30 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:45:50.221418 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=91,keystone:INSERT=1 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:46:48.970170 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7121f77-e615-4499-b59b-ecea3c937ced None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:46:48.971096 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7121f77-e615-4499-b59b-ecea3c937ced None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:46:48.971623 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7121f77-e615-4499-b59b-ecea3c937ced None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:46:48.977463 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 648/1307] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:46:48 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:46:48.978850 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15bcd510-40eb-4d9f-accb-ac96ddcd9475 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:46:48.979309 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15bcd510-40eb-4d9f-accb-ac96ddcd9475 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:46:48.979399 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15bcd510-40eb-4d9f-accb-ac96ddcd9475 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:46:49.082820 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-15bcd510-40eb-4d9f-accb-ac96ddcd9475 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:46:49.084075 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-15bcd510-40eb-4d9f-accb-ac96ddcd9475 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:46:49.251515 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 660/1308] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:46:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 273 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:46:49.552240 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee8457b0-2ba9-47c4-8f2f-c543414739f5 None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:46:49.552546 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee8457b0-2ba9-47c4-8f2f-c543414739f5 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:46:49.552782 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee8457b0-2ba9-47c4-8f2f-c543414739f5 None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:46:49.554376 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 649/1309] 10.222.0.42 () {54 vars in 965 bytes} [Sun Aug 28 00:46:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:46:49.561246 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f78543a7-38dd-4021-92cd-ab6628bb3e2e None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:46:49.561494 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f78543a7-38dd-4021-92cd-ab6628bb3e2e None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:46:49.561802 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f78543a7-38dd-4021-92cd-ab6628bb3e2e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:46:49.658949 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-f78543a7-38dd-4021-92cd-ab6628bb3e2e None None] MFA Rules not processed for user `364a28c108e744728917ba2b0aa2347d`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:46:49.660327 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-f78543a7-38dd-4021-92cd-ab6628bb3e2e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:46:49.686284 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:46:49.755972 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 661/1310] 10.222.0.42 () {58 vars in 1062 bytes} [Sun Aug 28 00:46:49 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 195 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:46:49.764259 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee8457b0-2ba9-47c4-8f2f-c543414739f5 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:46:49.766700 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-ee8457b0-2ba9-47c4-8f2f-c543414739f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:46:49.896744 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee8457b0-2ba9-47c4-8f2f-c543414739f5 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:46:49.976683 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c764a9ea-6e17-4d32-a9b0-6e212a4a4967 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:46:49.978721 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c764a9ea-6e17-4d32-a9b0-6e212a4a4967 service neutron] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:46:49.978721 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c764a9ea-6e17-4d32-a9b0-6e212a4a4967 service neutron] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:46:49.978721 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c764a9ea-6e17-4d32-a9b0-6e212a4a4967 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:46:49.979493 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-c764a9ea-6e17-4d32-a9b0-6e212a4a4967 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:46:50.048273 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:46:50.130411 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c764a9ea-6e17-4d32-a9b0-6e212a4a4967 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:46:50.132261 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c764a9ea-6e17-4d32-a9b0-6e212a4a4967 service neutron] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:46:50.156416 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 650/1311] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:46:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 392 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:46:52.377447 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c34e3998-cddb-407c-93d4-c11f984dea76 None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:46:52.377840 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c34e3998-cddb-407c-93d4-c11f984dea76 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:46:52.378014 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c34e3998-cddb-407c-93d4-c11f984dea76 None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:46:52.379699 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 662/1312] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:46:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:46:52.385475 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18b37d9d-f854-4690-920a-06342deadf5b None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:46:52.386022 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18b37d9d-f854-4690-920a-06342deadf5b None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:46:52.386022 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18b37d9d-f854-4690-920a-06342deadf5b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:46:52.488205 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-18b37d9d-f854-4690-920a-06342deadf5b None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:46:52.489485 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-18b37d9d-f854-4690-920a-06342deadf5b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:46:52.592894 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 651/1313] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:46:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 209 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:46:52.873546 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c34e3998-cddb-407c-93d4-c11f984dea76 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:46:52.875123 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-c34e3998-cddb-407c-93d4-c11f984dea76 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:46:52.988798 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c34e3998-cddb-407c-93d4-c11f984dea76 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:46:53.066090 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:46:53.069131 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b258441-23ac-4760-a372-1cfa6beebab5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:46:53.069781 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b258441-23ac-4760-a372-1cfa6beebab5 service neutron] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:46:53.070326 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b258441-23ac-4760-a372-1cfa6beebab5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:46:53.070645 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b258441-23ac-4760-a372-1cfa6beebab5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:46:53.073295 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-3b258441-23ac-4760-a372-1cfa6beebab5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:46:53.200698 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b258441-23ac-4760-a372-1cfa6beebab5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:46:53.202220 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b258441-23ac-4760-a372-1cfa6beebab5 service neutron] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:46:53.222926 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 663/1314] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:46:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 350 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:46:55.403733 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b245d86a-2ea9-4074-9cb7-6262fc1808e8 None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:46:55.404293 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b245d86a-2ea9-4074-9cb7-6262fc1808e8 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:46:55.404293 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b245d86a-2ea9-4074-9cb7-6262fc1808e8 None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:46:55.405694 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 652/1315] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:46:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:46:55.411175 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1248ad2-d7b1-491f-8373-3624c5e872ee None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:46:55.411615 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1248ad2-d7b1-491f-8373-3624c5e872ee None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:46:55.411709 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1248ad2-d7b1-491f-8373-3624c5e872ee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:46:55.482281 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-a1248ad2-d7b1-491f-8373-3624c5e872ee None None] MFA Rules not processed for user `196ac391d55640f99b97d658e79fbe0d`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:46:55.483519 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-a1248ad2-d7b1-491f-8373-3624c5e872ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:46:55.534675 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:46:55.583479 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 664/1316] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:46:55 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:46:55.863485 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b245d86a-2ea9-4074-9cb7-6262fc1808e8 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:46:55.865199 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-b245d86a-2ea9-4074-9cb7-6262fc1808e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:46:55.876333 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:46:55.977450 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b245d86a-2ea9-4074-9cb7-6262fc1808e8 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:46:56.055093 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92e0b69a-f3b3-422d-b8c0-a9ed1822bc86 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:46:56.055698 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92e0b69a-f3b3-422d-b8c0-a9ed1822bc86 service neutron] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:46:56.055977 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92e0b69a-f3b3-422d-b8c0-a9ed1822bc86 service neutron] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:46:56.056241 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92e0b69a-f3b3-422d-b8c0-a9ed1822bc86 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:46:56.057904 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-92e0b69a-f3b3-422d-b8c0-a9ed1822bc86 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:46:56.132134 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:46:56.179234 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92e0b69a-f3b3-422d-b8c0-a9ed1822bc86 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:46:56.180690 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92e0b69a-f3b3-422d-b8c0-a9ed1822bc86 service neutron] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:46:56.202178 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 653/1317] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:46:55 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 339 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:46:58.657445 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ce1f28-f01b-483a-b802-245957e6ff66 None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:46:58.658050 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ce1f28-f01b-483a-b802-245957e6ff66 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:46:58.658390 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ce1f28-f01b-483a-b802-245957e6ff66 None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:46:58.660892 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 665/1318] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:46:58 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:46:58.667359 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce4c49d9-453a-4655-b6bb-d8bed5c8f6ff None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:46:58.667820 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce4c49d9-453a-4655-b6bb-d8bed5c8f6ff None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:46:58.667914 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce4c49d9-453a-4655-b6bb-d8bed5c8f6ff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:46:58.735769 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-ce4c49d9-453a-4655-b6bb-d8bed5c8f6ff None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:46:58.737011 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-ce4c49d9-453a-4655-b6bb-d8bed5c8f6ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:46:58.847069 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 654/1319] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:46:58 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:46:59.148490 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-073d050e-e972-4f36-b2a9-701dfedcaf66 None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:46:59.148820 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-073d050e-e972-4f36-b2a9-701dfedcaf66 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:46:59.149053 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-073d050e-e972-4f36-b2a9-701dfedcaf66 None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:46:59.150392 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 666/1320] 10.222.0.42 () {54 vars in 965 bytes} [Sun Aug 28 00:46:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:46:59.156317 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b9b0944-7303-4b0e-92b4-bc846e6c5cbb None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:46:59.157247 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b9b0944-7303-4b0e-92b4-bc846e6c5cbb None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:46:59.157541 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b9b0944-7303-4b0e-92b4-bc846e6c5cbb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:46:59.180002 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:46:59.236477 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-7b9b0944-7303-4b0e-92b4-bc846e6c5cbb None None] MFA Rules not processed for user `364a28c108e744728917ba2b0aa2347d`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:46:59.237794 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-7b9b0944-7303-4b0e-92b4-bc846e6c5cbb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:46:59.320781 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 655/1321] 10.222.0.42 () {58 vars in 1062 bytes} [Sun Aug 28 00:46:59 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:46:59.330106 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-073d050e-e972-4f36-b2a9-701dfedcaf66 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:46:59.331650 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-073d050e-e972-4f36-b2a9-701dfedcaf66 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:46:59.447951 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-073d050e-e972-4f36-b2a9-701dfedcaf66 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:46:59.529510 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c2bbf4f-510b-4863-9552-e19ff90a206f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:46:59.530152 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c2bbf4f-510b-4863-9552-e19ff90a206f service neutron] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:46:59.530403 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c2bbf4f-510b-4863-9552-e19ff90a206f service neutron] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:46:59.530663 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c2bbf4f-510b-4863-9552-e19ff90a206f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:46:59.532383 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-8c2bbf4f-510b-4863-9552-e19ff90a206f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:46:59.544378 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:46:59.657815 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c2bbf4f-510b-4863-9552-e19ff90a206f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:46:59.659320 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c2bbf4f-510b-4863-9552-e19ff90a206f service neutron] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:46:59.678817 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 667/1322] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:46:59 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 349 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:47:02.737753 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c26aa3-43c4-4b61-93ed-63e07ec1e956 None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:47:02.738301 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c26aa3-43c4-4b61-93ed-63e07ec1e956 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:47:02.738755 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c26aa3-43c4-4b61-93ed-63e07ec1e956 None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:47:02.740606 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 656/1323] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:47:02 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:47:02.746248 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0b249d1-5c80-49d2-a3f4-084e467a360a None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:47:02.746736 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0b249d1-5c80-49d2-a3f4-084e467a360a None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:47:02.746818 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0b249d1-5c80-49d2-a3f4-084e467a360a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:47:02.817284 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-f0b249d1-5c80-49d2-a3f4-084e467a360a None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:47:02.818512 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-f0b249d1-5c80-49d2-a3f4-084e467a360a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:47:02.883256 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:47:02.923962 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 668/1324] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:47:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:47:03.208162 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50c26aa3-43c4-4b61-93ed-63e07ec1e956 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:47:03.209738 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-50c26aa3-43c4-4b61-93ed-63e07ec1e956 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:47:03.317003 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50c26aa3-43c4-4b61-93ed-63e07ec1e956 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:47:03.331857 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:47:03.399545 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6f0a7e4-7b0e-4677-a7da-f97e553c2c8c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:47:03.400147 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6f0a7e4-7b0e-4677-a7da-f97e553c2c8c service neutron] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:47:03.400430 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6f0a7e4-7b0e-4677-a7da-f97e553c2c8c service neutron] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:47:03.400704 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6f0a7e4-7b0e-4677-a7da-f97e553c2c8c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:47:03.402366 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-c6f0a7e4-7b0e-4677-a7da-f97e553c2c8c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:47:03.516405 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6f0a7e4-7b0e-4677-a7da-f97e553c2c8c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:47:03.517808 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6f0a7e4-7b0e-4677-a7da-f97e553c2c8c service neutron] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:47:03.538832 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 657/1325] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:47:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 331 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:47:05.567950 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d4bfe6c-62cf-4cec-a6b6-bb6838a814f1 None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:47:05.568532 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d4bfe6c-62cf-4cec-a6b6-bb6838a814f1 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:47:05.568532 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d4bfe6c-62cf-4cec-a6b6-bb6838a814f1 None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:47:05.569815 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 669/1326] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:47:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:47:05.576048 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a404762a-8eb7-4b67-aa8c-b12184db3897 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:47:05.576599 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a404762a-8eb7-4b67-aa8c-b12184db3897 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:47:05.576722 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a404762a-8eb7-4b67-aa8c-b12184db3897 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:47:05.630144 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:47:05.650071 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-a404762a-8eb7-4b67-aa8c-b12184db3897 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:47:05.651312 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-a404762a-8eb7-4b67-aa8c-b12184db3897 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:47:05.753403 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 658/1327] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:47:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:47:06.028411 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d4bfe6c-62cf-4cec-a6b6-bb6838a814f1 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:47:06.030024 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-0d4bfe6c-62cf-4cec-a6b6-bb6838a814f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:47:06.139406 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d4bfe6c-62cf-4cec-a6b6-bb6838a814f1 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:47:06.213824 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da57545f-532d-4ba2-b6d1-74b396ea5ac2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:47:06.214444 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da57545f-532d-4ba2-b6d1-74b396ea5ac2 service neutron] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:47:06.214724 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da57545f-532d-4ba2-b6d1-74b396ea5ac2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:47:06.214997 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da57545f-532d-4ba2-b6d1-74b396ea5ac2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:47:06.216628 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-da57545f-532d-4ba2-b6d1-74b396ea5ac2 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:47:06.234364 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:47:06.335944 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da57545f-532d-4ba2-b6d1-74b396ea5ac2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:47:06.337317 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da57545f-532d-4ba2-b6d1-74b396ea5ac2 service neutron] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:47:06.356040 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 670/1328] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:47:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 328 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:47:08.301874 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a35db55-3764-48d9-bf28-a6d45d1da243 None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:47:08.302466 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a35db55-3764-48d9-bf28-a6d45d1da243 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:47:08.302466 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a35db55-3764-48d9-bf28-a6d45d1da243 None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:47:08.303811 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 659/1329] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:47:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:47:08.309397 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09714062-d981-4f9b-858d-a5bd94f45176 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:47:08.309879 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09714062-d981-4f9b-858d-a5bd94f45176 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:47:08.309993 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09714062-d981-4f9b-858d-a5bd94f45176 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:47:08.380354 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-09714062-d981-4f9b-858d-a5bd94f45176 None None] MFA Rules not processed for user `196ac391d55640f99b97d658e79fbe0d`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:47:08.381615 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-09714062-d981-4f9b-858d-a5bd94f45176 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:47:08.449960 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:47:08.484917 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 671/1330] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:47:08 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:47:08.763790 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a35db55-3764-48d9-bf28-a6d45d1da243 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:47:08.765370 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-0a35db55-3764-48d9-bf28-a6d45d1da243 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:47:08.876418 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a35db55-3764-48d9-bf28-a6d45d1da243 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:47:08.897946 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:47:08.958869 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0627f17d-fdd8-4e5a-b3d1-8d6f58ebcc07 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:47:08.959415 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0627f17d-fdd8-4e5a-b3d1-8d6f58ebcc07 service neutron] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:47:08.959691 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0627f17d-fdd8-4e5a-b3d1-8d6f58ebcc07 service neutron] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:47:08.959960 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0627f17d-fdd8-4e5a-b3d1-8d6f58ebcc07 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:47:08.961679 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-0627f17d-fdd8-4e5a-b3d1-8d6f58ebcc07 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:47:09.079292 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0627f17d-fdd8-4e5a-b3d1-8d6f58ebcc07 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:47:09.080855 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0627f17d-fdd8-4e5a-b3d1-8d6f58ebcc07 service neutron] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:47:09.100233 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 660/1331] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:47:08 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 337 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:47:11.251384 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1a00b6-f6b8-4457-a2c4-5691607959bf None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:47:11.252007 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1a00b6-f6b8-4457-a2c4-5691607959bf None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:47:11.252007 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1a00b6-f6b8-4457-a2c4-5691607959bf None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:47:11.253392 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 672/1332] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:47:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:47:11.258804 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d05306c-80ad-4457-b200-666af5314864 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:47:11.259233 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d05306c-80ad-4457-b200-666af5314864 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:47:11.259350 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d05306c-80ad-4457-b200-666af5314864 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:47:11.323143 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:47:11.327207 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-5d05306c-80ad-4457-b200-666af5314864 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:47:11.328994 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-5d05306c-80ad-4457-b200-666af5314864 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:47:11.425966 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 661/1333] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:47:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:47:11.705960 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d1a00b6-f6b8-4457-a2c4-5691607959bf None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:47:11.707543 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-3d1a00b6-f6b8-4457-a2c4-5691607959bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:47:11.817897 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d1a00b6-f6b8-4457-a2c4-5691607959bf None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:47:11.893387 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a672651-623e-457b-bf96-bc4243541936 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:47:11.894001 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a672651-623e-457b-bf96-bc4243541936 service neutron] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:47:11.894273 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a672651-623e-457b-bf96-bc4243541936 service neutron] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:47:11.894546 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a672651-623e-457b-bf96-bc4243541936 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:47:11.896153 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-0a672651-623e-457b-bf96-bc4243541936 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:47:11.923864 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:47:12.010505 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a672651-623e-457b-bf96-bc4243541936 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:47:12.011865 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a672651-623e-457b-bf96-bc4243541936 service neutron] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:47:12.030343 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 673/1334] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:47:11 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 325 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:47:14.027658 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f60433b-0142-4a60-8a24-9975feff8008 None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:47:14.028241 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f60433b-0142-4a60-8a24-9975feff8008 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:47:14.028241 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f60433b-0142-4a60-8a24-9975feff8008 None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:47:14.029599 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 662/1335] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:47:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:47:14.035237 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1953826-acdc-4eef-87b5-db083231ac05 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:47:14.035734 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1953826-acdc-4eef-87b5-db083231ac05 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:47:14.035840 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1953826-acdc-4eef-87b5-db083231ac05 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:47:14.106672 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-c1953826-acdc-4eef-87b5-db083231ac05 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:47:14.107905 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-c1953826-acdc-4eef-87b5-db083231ac05 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:47:14.187381 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:47:14.211317 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 674/1336] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:47:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:47:14.490223 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f60433b-0142-4a60-8a24-9975feff8008 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:47:14.491802 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-7f60433b-0142-4a60-8a24-9975feff8008 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:47:14.600651 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f60433b-0142-4a60-8a24-9975feff8008 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:47:14.631793 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:47:14.681199 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89da8dac-903f-4e62-9014-ab60cffed7af service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:47:14.681826 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89da8dac-903f-4e62-9014-ab60cffed7af service neutron] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:47:14.682114 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89da8dac-903f-4e62-9014-ab60cffed7af service neutron] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:47:14.682385 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89da8dac-903f-4e62-9014-ab60cffed7af service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:47:14.684115 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-89da8dac-903f-4e62-9014-ab60cffed7af service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:47:14.803590 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89da8dac-903f-4e62-9014-ab60cffed7af service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:47:14.804993 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89da8dac-903f-4e62-9014-ab60cffed7af service neutron] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:47:14.825397 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 663/1337] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:47:14 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 336 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:47:17.176748 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5ff907e-cf4b-49d0-a739-89c4bafe8ea0 None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:47:17.177338 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5ff907e-cf4b-49d0-a739-89c4bafe8ea0 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:47:17.177338 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5ff907e-cf4b-49d0-a739-89c4bafe8ea0 None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:47:17.178802 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 675/1338] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:47:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:47:17.184505 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abfc29d6-6759-4ac1-96c4-ce8b38aa6f7b None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:47:17.185000 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abfc29d6-6759-4ac1-96c4-ce8b38aa6f7b None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:47:17.185117 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abfc29d6-6759-4ac1-96c4-ce8b38aa6f7b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:47:17.257815 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-abfc29d6-6759-4ac1-96c4-ce8b38aa6f7b None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:47:17.259034 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-abfc29d6-6759-4ac1-96c4-ce8b38aa6f7b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:47:17.267632 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:47:17.374757 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 664/1339] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:47:17 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 191 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:47:17.652387 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5ff907e-cf4b-49d0-a739-89c4bafe8ea0 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:47:17.653994 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-f5ff907e-cf4b-49d0-a739-89c4bafe8ea0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:47:17.765600 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5ff907e-cf4b-49d0-a739-89c4bafe8ea0 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:47:17.843167 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-554a13a5-2f3b-43f1-b496-5df762bd8ec9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:47:17.843774 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-554a13a5-2f3b-43f1-b496-5df762bd8ec9 service neutron] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:47:17.844046 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-554a13a5-2f3b-43f1-b496-5df762bd8ec9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:47:17.844311 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-554a13a5-2f3b-43f1-b496-5df762bd8ec9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:47:17.845966 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-554a13a5-2f3b-43f1-b496-5df762bd8ec9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:47:17.882674 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:47:17.974695 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-554a13a5-2f3b-43f1-b496-5df762bd8ec9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:47:17.976114 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-554a13a5-2f3b-43f1-b496-5df762bd8ec9 service neutron] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:47:17.996510 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 676/1340] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:47:17 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 345 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:47:21.942324 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be56209f-3d9b-423e-ba0a-a468d4216696 None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:47:21.942930 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be56209f-3d9b-423e-ba0a-a468d4216696 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:47:21.943269 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be56209f-3d9b-423e-ba0a-a468d4216696 None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:47:21.945150 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 665/1341] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:47:21 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:47:21.951949 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19a22a85-382e-4af1-befb-93c8eb50c3e2 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:47:21.952824 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19a22a85-382e-4af1-befb-93c8eb50c3e2 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:47:21.953041 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19a22a85-382e-4af1-befb-93c8eb50c3e2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:47:22.038101 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-19a22a85-382e-4af1-befb-93c8eb50c3e2 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:47:22.039482 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-19a22a85-382e-4af1-befb-93c8eb50c3e2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:47:22.175326 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:47:22.191863 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 677/1342] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:47:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 241 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:47:22.490838 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be56209f-3d9b-423e-ba0a-a468d4216696 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:47:22.492387 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-be56209f-3d9b-423e-ba0a-a468d4216696 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:47:22.605028 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be56209f-3d9b-423e-ba0a-a468d4216696 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:47:22.646481 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:47:22.690982 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35307740-2467-43ab-9d1b-c14b0f207663 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:47:22.691596 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35307740-2467-43ab-9d1b-c14b0f207663 service neutron] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:47:22.691868 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35307740-2467-43ab-9d1b-c14b0f207663 service neutron] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:47:22.692131 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35307740-2467-43ab-9d1b-c14b0f207663 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:47:22.693806 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-35307740-2467-43ab-9d1b-c14b0f207663 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:47:22.834514 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35307740-2467-43ab-9d1b-c14b0f207663 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:47:22.835906 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35307740-2467-43ab-9d1b-c14b0f207663 service neutron] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:47:22.861079 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 666/1343] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:47:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 371 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:47:25.429796 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e108a1b-c0f7-4721-a64b-6f341df03441 None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:47:25.430545 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e108a1b-c0f7-4721-a64b-6f341df03441 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:47:25.430545 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e108a1b-c0f7-4721-a64b-6f341df03441 None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:47:25.431685 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 678/1344] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:47:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:47:25.438166 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e66d76d2-1b39-4909-8d41-c2cd227bf3b8 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:47:25.438654 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e66d76d2-1b39-4909-8d41-c2cd227bf3b8 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:47:25.439036 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e66d76d2-1b39-4909-8d41-c2cd227bf3b8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:47:25.521539 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-e66d76d2-1b39-4909-8d41-c2cd227bf3b8 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:47:25.522784 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-e66d76d2-1b39-4909-8d41-c2cd227bf3b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:47:25.543653 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:47:25.653807 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 667/1345] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:47:25 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 218 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:47:25.942664 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e108a1b-c0f7-4721-a64b-6f341df03441 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:47:25.944614 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-7e108a1b-c0f7-4721-a64b-6f341df03441 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:47:26.066672 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e108a1b-c0f7-4721-a64b-6f341df03441 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:47:26.168322 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4134b8e7-b020-4141-9e07-e6491b7baad1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:47:26.169455 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4134b8e7-b020-4141-9e07-e6491b7baad1 service neutron] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:47:26.169811 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4134b8e7-b020-4141-9e07-e6491b7baad1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:47:26.170289 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4134b8e7-b020-4141-9e07-e6491b7baad1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:47:26.172998 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-4134b8e7-b020-4141-9e07-e6491b7baad1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:47:26.234761 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:47:26.319773 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4134b8e7-b020-4141-9e07-e6491b7baad1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:47:26.321357 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4134b8e7-b020-4141-9e07-e6491b7baad1 service neutron] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:47:26.341657 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 679/1346] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:47:25 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 400 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:47:30.510418 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7e7a59c-17a7-482d-96c6-4a5bfa1c58b8 None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:47:30.511302 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7e7a59c-17a7-482d-96c6-4a5bfa1c58b8 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:47:30.511302 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7e7a59c-17a7-482d-96c6-4a5bfa1c58b8 None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:47:30.513321 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 668/1347] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:47:30 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:47:30.519520 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5ed9671-d2b9-4520-a264-40d85b0b0222 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:47:30.520082 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5ed9671-d2b9-4520-a264-40d85b0b0222 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:47:30.520082 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5ed9671-d2b9-4520-a264-40d85b0b0222 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:47:30.599153 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-a5ed9671-d2b9-4520-a264-40d85b0b0222 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:47:30.600178 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-a5ed9671-d2b9-4520-a264-40d85b0b0222 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:47:30.728043 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:47:30.740188 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 680/1348] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:47:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 222 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 28 00:47:31.014676 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7e7a59c-17a7-482d-96c6-4a5bfa1c58b8 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:47:31.016280 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-b7e7a59c-17a7-482d-96c6-4a5bfa1c58b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:47:31.134236 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7e7a59c-17a7-482d-96c6-4a5bfa1c58b8 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:47:31.182314 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:47:31.217255 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7da0b1b0-997f-4a71-ba85-0a85a68be2a6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:47:31.218030 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7da0b1b0-997f-4a71-ba85-0a85a68be2a6 service neutron] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:47:31.218402 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7da0b1b0-997f-4a71-ba85-0a85a68be2a6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:47:31.218706 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7da0b1b0-997f-4a71-ba85-0a85a68be2a6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:47:31.220565 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-7da0b1b0-997f-4a71-ba85-0a85a68be2a6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:47:31.337982 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7da0b1b0-997f-4a71-ba85-0a85a68be2a6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:47:31.339332 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7da0b1b0-997f-4a71-ba85-0a85a68be2a6 service neutron] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:47:31.358237 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 669/1349] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:47:31 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 344 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:47:36.309254 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e6c462e-51f7-47e1-ac5f-aa228427b1fa None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:47:36.310885 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e6c462e-51f7-47e1-ac5f-aa228427b1fa None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:47:36.311844 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e6c462e-51f7-47e1-ac5f-aa228427b1fa None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:47:36.313725 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 681/1350] 10.222.0.42 () {54 vars in 921 bytes} [Sun Aug 28 00:47:36 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:47:36.319419 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8af2a38-5c6b-4c8c-b8b3-ad5837ce16fc None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:47:36.319900 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8af2a38-5c6b-4c8c-b8b3-ad5837ce16fc None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:47:36.319985 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8af2a38-5c6b-4c8c-b8b3-ad5837ce16fc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:47:36.419679 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-f8af2a38-5c6b-4c8c-b8b3-ad5837ce16fc None None] MFA Rules not processed for user `c2a3834fa6894dada3291e4781bbd553`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:47:36.421575 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-f8af2a38-5c6b-4c8c-b8b3-ad5837ce16fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:47:36.444516 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:47:36.591055 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 670/1351] 10.222.0.42 () {58 vars in 1018 bytes} [Sun Aug 28 00:47:36 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 272 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:47:40.666077 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7d5bbc1-baad-41ee-ba09-7b9477689039 None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:47:40.666651 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7d5bbc1-baad-41ee-ba09-7b9477689039 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:47:40.666651 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7d5bbc1-baad-41ee-ba09-7b9477689039 None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:47:40.668227 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 682/1352] 10.222.0.42 () {54 vars in 921 bytes} [Sun Aug 28 00:47:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:47:40.673654 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93c8faa0-c73a-45c3-8061-3bbf111b8b8b None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:47:40.674145 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93c8faa0-c73a-45c3-8061-3bbf111b8b8b None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:47:40.674145 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93c8faa0-c73a-45c3-8061-3bbf111b8b8b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:47:40.754980 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-93c8faa0-c73a-45c3-8061-3bbf111b8b8b None None] MFA Rules not processed for user `c2a3834fa6894dada3291e4781bbd553`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:47:40.756204 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-93c8faa0-c73a-45c3-8061-3bbf111b8b8b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:47:40.829766 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:47:40.869640 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 671/1353] 10.222.0.42 () {58 vars in 1018 bytes} [Sun Aug 28 00:47:40 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 197 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:47:42.358340 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-152d5f85-fdf2-4473-885f-b483799dedb8 None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:47:42.359004 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-152d5f85-fdf2-4473-885f-b483799dedb8 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:47:42.359004 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-152d5f85-fdf2-4473-885f-b483799dedb8 None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:47:42.360209 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 683/1354] 10.222.0.42 () {54 vars in 921 bytes} [Sun Aug 28 00:47:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:47:42.365599 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf9fdfa2-1e49-4ea6-8c63-358fca3b6a89 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:47:42.366067 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf9fdfa2-1e49-4ea6-8c63-358fca3b6a89 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:47:42.366285 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf9fdfa2-1e49-4ea6-8c63-358fca3b6a89 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:47:42.449375 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-cf9fdfa2-1e49-4ea6-8c63-358fca3b6a89 None None] MFA Rules not processed for user `c2a3834fa6894dada3291e4781bbd553`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:47:42.450622 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-cf9fdfa2-1e49-4ea6-8c63-358fca3b6a89 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:47:42.569224 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 672/1355] 10.222.0.42 () {58 vars in 1018 bytes} [Sun Aug 28 00:47:42 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 205 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:47:45.837153 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9afc9b3d-bad4-4ec1-80af-860750cb8ce0 None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:47:45.837683 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9afc9b3d-bad4-4ec1-80af-860750cb8ce0 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:47:45.837934 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9afc9b3d-bad4-4ec1-80af-860750cb8ce0 None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:47:45.839389 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 684/1356] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:47:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:47:45.845830 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fde4e8e8-54e2-4d01-acf0-99b5aefc0596 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:47:45.846280 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fde4e8e8-54e2-4d01-acf0-99b5aefc0596 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:47:45.846373 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fde4e8e8-54e2-4d01-acf0-99b5aefc0596 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:47:45.885240 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:47:45.930950 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-fde4e8e8-54e2-4d01-acf0-99b5aefc0596 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:47:45.932135 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-fde4e8e8-54e2-4d01-acf0-99b5aefc0596 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:47:46.060104 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 673/1357] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:47:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 215 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:47:46.358591 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53521457-f389-4709-a09b-435ffa1e2e52 None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:47:46.358856 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53521457-f389-4709-a09b-435ffa1e2e52 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:47:46.359316 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53521457-f389-4709-a09b-435ffa1e2e52 None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:47:46.360703 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 685/1358] 10.222.0.42 () {54 vars in 962 bytes} [Sun Aug 28 00:47:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:47:46.369557 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a0d464b-88e6-4045-a131-8c77b795211a None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:47:46.370242 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a0d464b-88e6-4045-a131-8c77b795211a None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:47:46.371257 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a0d464b-88e6-4045-a131-8c77b795211a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:47:46.460569 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-4a0d464b-88e6-4045-a131-8c77b795211a None None] MFA Rules not processed for user `168b07df50c24593ba73dbde5b037aed`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:47:46.462190 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-4a0d464b-88e6-4045-a131-8c77b795211a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:47:46.513839 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:47:46.592596 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 674/1359] 10.222.0.42 () {58 vars in 1059 bytes} [Sun Aug 28 00:47:46 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 224 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:47:46.600542 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53521457-f389-4709-a09b-435ffa1e2e52 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:47:46.602537 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-53521457-f389-4709-a09b-435ffa1e2e52 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:47:46.768270 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53521457-f389-4709-a09b-435ffa1e2e52 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:47:46.851994 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:47:46.900624 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-347a2f27-0c42-4123-ba89-3f24a05b6e7c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '168b07df50c24593ba73dbde5b037aed', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:47:46.901315 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-347a2f27-0c42-4123-ba89-3f24a05b6e7c service nova] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:47:46.901881 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-347a2f27-0c42-4123-ba89-3f24a05b6e7c service nova] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:47:46.902161 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-347a2f27-0c42-4123-ba89-3f24a05b6e7c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:47:46.903934 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-347a2f27-0c42-4123-ba89-3f24a05b6e7c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:47:47.007381 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-347a2f27-0c42-4123-ba89-3f24a05b6e7c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:47:47.009005 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-347a2f27-0c42-4123-ba89-3f24a05b6e7c service nova] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:47:47.029014 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 686/1360] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:47:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 429 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:47:49.280760 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a853dffc-1c4c-4e40-ac5b-eb52ea51bf05 None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:47:49.281355 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a853dffc-1c4c-4e40-ac5b-eb52ea51bf05 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:47:49.281834 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a853dffc-1c4c-4e40-ac5b-eb52ea51bf05 None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:47:49.283379 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 675/1361] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:47:49 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:47:49.288890 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94eafa12-5c7e-42fd-be29-908b8e3cce31 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:47:49.289427 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94eafa12-5c7e-42fd-be29-908b8e3cce31 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:47:49.289427 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94eafa12-5c7e-42fd-be29-908b8e3cce31 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:47:49.361668 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-94eafa12-5c7e-42fd-be29-908b8e3cce31 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:47:49.362860 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-94eafa12-5c7e-42fd-be29-908b8e3cce31 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:47:49.425144 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:47:49.484778 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 687/1362] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:47:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 197 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:47:49.775379 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a853dffc-1c4c-4e40-ac5b-eb52ea51bf05 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:47:49.776940 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-a853dffc-1c4c-4e40-ac5b-eb52ea51bf05 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:47:49.948581 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a853dffc-1c4c-4e40-ac5b-eb52ea51bf05 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:47:49.966802 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:47:50.100259 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ca39938-5519-4b6f-9c89-8c984fe4c7f7 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '168b07df50c24593ba73dbde5b037aed', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:47:50.100878 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca39938-5519-4b6f-9c89-8c984fe4c7f7 service nova] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:47:50.101197 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca39938-5519-4b6f-9c89-8c984fe4c7f7 service nova] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:47:50.101466 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca39938-5519-4b6f-9c89-8c984fe4c7f7 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:47:50.103145 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-2ca39938-5519-4b6f-9c89-8c984fe4c7f7 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:47:50.217162 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ca39938-5519-4b6f-9c89-8c984fe4c7f7 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:47:50.218855 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ca39938-5519-4b6f-9c89-8c984fe4c7f7 service nova] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:47:50.246112 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 676/1363] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:47:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 471 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:47:52.517932 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-046803a9-bb25-4b31-9d11-8181a0af4317 None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:47:52.518513 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-046803a9-bb25-4b31-9d11-8181a0af4317 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:47:52.518513 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-046803a9-bb25-4b31-9d11-8181a0af4317 None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:47:52.519836 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 688/1364] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:47:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:47:52.525542 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff1b6da-b331-40e5-a718-fc5d0845b229 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:47:52.526009 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff1b6da-b331-40e5-a718-fc5d0845b229 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:47:52.526088 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff1b6da-b331-40e5-a718-fc5d0845b229 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:47:52.543189 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:47:52.599425 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-fff1b6da-b331-40e5-a718-fc5d0845b229 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:47:52.600660 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-fff1b6da-b331-40e5-a718-fc5d0845b229 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:47:52.705401 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 677/1365] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:47:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:47:52.994596 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3fb9ebe-243e-41d2-8c8d-34bd99a77bca None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:47:52.995263 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3fb9ebe-243e-41d2-8c8d-34bd99a77bca None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:47:52.995630 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3fb9ebe-243e-41d2-8c8d-34bd99a77bca None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:47:52.997226 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 689/1366] 10.222.0.42 () {54 vars in 962 bytes} [Sun Aug 28 00:47:52 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:47:53.003478 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55d5ff50-e548-417f-a83c-1f5b15c82e9b None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:47:53.003721 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55d5ff50-e548-417f-a83c-1f5b15c82e9b None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:47:53.004211 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55d5ff50-e548-417f-a83c-1f5b15c82e9b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:47:53.217817 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-55d5ff50-e548-417f-a83c-1f5b15c82e9b None None] MFA Rules not processed for user `168b07df50c24593ba73dbde5b037aed`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:47:53.219054 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-55d5ff50-e548-417f-a83c-1f5b15c82e9b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:47:53.268341 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:47:53.385336 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 678/1367] 10.222.0.42 () {58 vars in 1059 bytes} [Sun Aug 28 00:47:53 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 383 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:47:53.388997 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3fb9ebe-243e-41d2-8c8d-34bd99a77bca None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:47:53.390602 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-f3fb9ebe-243e-41d2-8c8d-34bd99a77bca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:47:53.549831 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3fb9ebe-243e-41d2-8c8d-34bd99a77bca None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:47:53.572645 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:47:53.674629 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a3fa207-866f-4194-8ddd-97a7d2296d84 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '168b07df50c24593ba73dbde5b037aed', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:47:53.675295 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a3fa207-866f-4194-8ddd-97a7d2296d84 service nova] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:47:53.675571 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a3fa207-866f-4194-8ddd-97a7d2296d84 service nova] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:47:53.675838 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a3fa207-866f-4194-8ddd-97a7d2296d84 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:47:53.677658 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-7a3fa207-866f-4194-8ddd-97a7d2296d84 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:47:53.795874 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a3fa207-866f-4194-8ddd-97a7d2296d84 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:47:53.797620 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a3fa207-866f-4194-8ddd-97a7d2296d84 service nova] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:47:53.821683 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 690/1368] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:47:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 433 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:47:54.124330 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-460eafea-94cb-4634-9cc2-722a7efa8483 None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:47:54.124810 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-460eafea-94cb-4634-9cc2-722a7efa8483 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:47:54.124929 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-460eafea-94cb-4634-9cc2-722a7efa8483 None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:47:54.126467 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 679/1369] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:47:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:47:54.131974 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a43a7d-938b-44a0-ae20-9135a224b96e None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:47:54.132259 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a43a7d-938b-44a0-ae20-9135a224b96e None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:47:54.132529 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a43a7d-938b-44a0-ae20-9135a224b96e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:47:54.169777 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:47:54.218051 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-b2a43a7d-938b-44a0-ae20-9135a224b96e None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:47:54.219539 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-b2a43a7d-938b-44a0-ae20-9135a224b96e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:47:54.357035 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 691/1370] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:47:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 223 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:47:54.933887 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-732764e9-a0b6-40a3-badd-7868e9dedee1 None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:47:54.933887 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-732764e9-a0b6-40a3-badd-7868e9dedee1 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:47:54.933887 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-732764e9-a0b6-40a3-badd-7868e9dedee1 None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:47:54.936789 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 680/1371] 10.222.0.42 () {54 vars in 964 bytes} [Sun Aug 28 00:47:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:47:54.940419 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aedc7d2-2b99-4bd8-8236-4b9a5317d114 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:47:54.941055 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aedc7d2-2b99-4bd8-8236-4b9a5317d114 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:47:54.941055 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aedc7d2-2b99-4bd8-8236-4b9a5317d114 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:47:55.051265 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-2aedc7d2-2b99-4bd8-8236-4b9a5317d114 None None] MFA Rules not processed for user `879afc99ebb9402b94fadc6091040c5c`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:47:55.051265 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-2aedc7d2-2b99-4bd8-8236-4b9a5317d114 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:47:55.103953 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:47:55.146111 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 692/1372] 10.222.0.42 () {58 vars in 1061 bytes} [Sun Aug 28 00:47:54 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 206 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:47:55.153559 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-732764e9-a0b6-40a3-badd-7868e9dedee1 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:47:55.155429 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-732764e9-a0b6-40a3-badd-7868e9dedee1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:47:55.268227 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-732764e9-a0b6-40a3-badd-7868e9dedee1 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:47:55.314249 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:47:55.360373 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80be9074-279e-4b0f-b09a-3c6ed16a1805 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:47:55.361007 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80be9074-279e-4b0f-b09a-3c6ed16a1805 service cinder] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:47:55.361343 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80be9074-279e-4b0f-b09a-3c6ed16a1805 service cinder] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:47:55.361556 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80be9074-279e-4b0f-b09a-3c6ed16a1805 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:47:55.363661 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-80be9074-279e-4b0f-b09a-3c6ed16a1805 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:47:55.511150 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80be9074-279e-4b0f-b09a-3c6ed16a1805 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:47:55.512899 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80be9074-279e-4b0f-b09a-3c6ed16a1805 service cinder] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:47:55.538895 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 681/1373] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:47:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 385 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:47:55.867286 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a35a39f-8dcb-43cd-bb03-5bdf56605c21 None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:47:55.867757 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a35a39f-8dcb-43cd-bb03-5bdf56605c21 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:47:55.868127 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a35a39f-8dcb-43cd-bb03-5bdf56605c21 None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:47:55.869908 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 693/1374] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:47:55 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:47:55.876863 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b10254b-bef6-4418-a2d0-affa7f45c812 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:47:55.877766 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b10254b-bef6-4418-a2d0-affa7f45c812 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:47:55.877766 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b10254b-bef6-4418-a2d0-affa7f45c812 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:47:55.959226 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-3b10254b-bef6-4418-a2d0-affa7f45c812 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:47:55.960442 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-3b10254b-bef6-4418-a2d0-affa7f45c812 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:47:55.984287 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:47:56.065069 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 682/1375] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:47:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:47:56.345582 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a35a39f-8dcb-43cd-bb03-5bdf56605c21 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:47:56.347106 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-1a35a39f-8dcb-43cd-bb03-5bdf56605c21 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:47:56.496130 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a35a39f-8dcb-43cd-bb03-5bdf56605c21 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:47:56.551448 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:47:56.614775 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbe3b890-10a7-4ce7-9f7a-07d3f88765f1 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '168b07df50c24593ba73dbde5b037aed', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:47:56.615388 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbe3b890-10a7-4ce7-9f7a-07d3f88765f1 service nova] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:47:56.615636 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbe3b890-10a7-4ce7-9f7a-07d3f88765f1 service nova] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:47:56.615921 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbe3b890-10a7-4ce7-9f7a-07d3f88765f1 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:47:56.617585 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-fbe3b890-10a7-4ce7-9f7a-07d3f88765f1 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:47:56.708915 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbe3b890-10a7-4ce7-9f7a-07d3f88765f1 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:47:56.710563 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbe3b890-10a7-4ce7-9f7a-07d3f88765f1 service nova] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:47:56.729568 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 694/1376] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:47:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 384 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:47:58.651824 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6428eec7-f951-4e96-98bd-18ac74f75d19 None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:47:58.652353 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6428eec7-f951-4e96-98bd-18ac74f75d19 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:47:58.652353 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6428eec7-f951-4e96-98bd-18ac74f75d19 None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:47:58.653961 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 683/1377] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:47:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:47:58.659461 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dab19f30-32d7-4c87-ae5e-5091ed0d1777 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:47:58.659899 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dab19f30-32d7-4c87-ae5e-5091ed0d1777 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:47:58.660003 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dab19f30-32d7-4c87-ae5e-5091ed0d1777 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:47:58.734371 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-dab19f30-32d7-4c87-ae5e-5091ed0d1777 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:47:58.735625 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-dab19f30-32d7-4c87-ae5e-5091ed0d1777 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:47:58.742580 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:47:58.848084 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 695/1378] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:47:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:47:59.157397 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6428eec7-f951-4e96-98bd-18ac74f75d19 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:47:59.158904 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-6428eec7-f951-4e96-98bd-18ac74f75d19 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:47:59.329562 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6428eec7-f951-4e96-98bd-18ac74f75d19 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:47:59.333735 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:47:59.454464 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-300c2278-6e5f-4f06-ad42-819c4a296d29 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '168b07df50c24593ba73dbde5b037aed', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:47:59.455242 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-300c2278-6e5f-4f06-ad42-819c4a296d29 service nova] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:47:59.455623 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-300c2278-6e5f-4f06-ad42-819c4a296d29 service nova] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:47:59.456018 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-300c2278-6e5f-4f06-ad42-819c4a296d29 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:47:59.457801 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-300c2278-6e5f-4f06-ad42-819c4a296d29 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:47:59.555055 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-300c2278-6e5f-4f06-ad42-819c4a296d29 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:47:59.556823 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-300c2278-6e5f-4f06-ad42-819c4a296d29 service nova] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:47:59.576691 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 684/1379] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:47:59 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 420 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:48:01.541339 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a889f104-7eb4-4356-9b70-ddf412abacf3 None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:48:01.542165 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a889f104-7eb4-4356-9b70-ddf412abacf3 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:48:01.542165 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a889f104-7eb4-4356-9b70-ddf412abacf3 None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:48:01.543355 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 696/1380] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:48:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:48:01.549188 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bd12d41-c651-49c8-809b-1118c9775438 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:48:01.549965 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bd12d41-c651-49c8-809b-1118c9775438 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:48:01.549965 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bd12d41-c651-49c8-809b-1118c9775438 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:48:01.561734 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:48:01.625757 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-8bd12d41-c651-49c8-809b-1118c9775438 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:48:01.627172 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-8bd12d41-c651-49c8-809b-1118c9775438 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:48:01.760070 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 685/1381] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:48:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 212 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:48:02.039025 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a889f104-7eb4-4356-9b70-ddf412abacf3 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:48:02.040552 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-a889f104-7eb4-4356-9b70-ddf412abacf3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:48:02.202816 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:48:02.214998 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a889f104-7eb4-4356-9b70-ddf412abacf3 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:48:02.342656 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f7ad727-765c-47e0-82b3-5e161d2851dd service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '168b07df50c24593ba73dbde5b037aed', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:48:02.343221 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f7ad727-765c-47e0-82b3-5e161d2851dd service nova] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:48:02.343493 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f7ad727-765c-47e0-82b3-5e161d2851dd service nova] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:48:02.343760 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f7ad727-765c-47e0-82b3-5e161d2851dd service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:48:02.345484 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-1f7ad727-765c-47e0-82b3-5e161d2851dd service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:48:02.440742 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f7ad727-765c-47e0-82b3-5e161d2851dd service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:48:02.442526 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f7ad727-765c-47e0-82b3-5e161d2851dd service nova] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:48:02.456471 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:48:02.474030 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 697/1382] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:48:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 435 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:48:04.427980 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7152a950-aafe-4f42-b7e2-3c1e5a8e4110 None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:48:04.428870 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7152a950-aafe-4f42-b7e2-3c1e5a8e4110 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:48:04.429201 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7152a950-aafe-4f42-b7e2-3c1e5a8e4110 None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:48:04.430719 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 686/1383] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:48:04 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:48:04.436499 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc825715-a023-4d2d-b4f2-df46394b9eef None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:48:04.437034 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc825715-a023-4d2d-b4f2-df46394b9eef None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:48:04.437130 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc825715-a023-4d2d-b4f2-df46394b9eef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:48:04.529508 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-fc825715-a023-4d2d-b4f2-df46394b9eef None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:48:04.531029 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-fc825715-a023-4d2d-b4f2-df46394b9eef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:48:04.658161 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 698/1384] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:48:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 223 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:48:04.944204 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7152a950-aafe-4f42-b7e2-3c1e5a8e4110 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:48:04.945997 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-7152a950-aafe-4f42-b7e2-3c1e5a8e4110 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:48:05.068207 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:48:05.160144 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7152a950-aafe-4f42-b7e2-3c1e5a8e4110 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:48:05.300359 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bb1af7f-c570-45ff-a1b7-77a8aee206bc service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '168b07df50c24593ba73dbde5b037aed', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:48:05.301117 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bb1af7f-c570-45ff-a1b7-77a8aee206bc service nova] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:48:05.301463 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bb1af7f-c570-45ff-a1b7-77a8aee206bc service nova] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:48:05.301816 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bb1af7f-c570-45ff-a1b7-77a8aee206bc service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:48:05.303850 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-6bb1af7f-c570-45ff-a1b7-77a8aee206bc service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:48:05.368255 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:48:05.416738 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bb1af7f-c570-45ff-a1b7-77a8aee206bc service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:48:05.418887 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bb1af7f-c570-45ff-a1b7-77a8aee206bc service nova] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:48:05.440082 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 687/1385] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:48:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 497 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:48:07.561890 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01e1747f-2e0e-47e7-b624-70dfcf1efe69 None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:48:07.562626 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01e1747f-2e0e-47e7-b624-70dfcf1efe69 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:48:07.562626 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01e1747f-2e0e-47e7-b624-70dfcf1efe69 None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:48:07.563861 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 699/1386] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:48:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:48:07.569961 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a384da7-b50e-442a-bf63-f3b5880afcfd None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:48:07.570644 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a384da7-b50e-442a-bf63-f3b5880afcfd None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:48:07.570644 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a384da7-b50e-442a-bf63-f3b5880afcfd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:48:07.670485 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-2a384da7-b50e-442a-bf63-f3b5880afcfd None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:48:07.671792 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-2a384da7-b50e-442a-bf63-f3b5880afcfd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:48:07.819660 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 688/1387] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:48:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 250 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 28 00:48:08.135866 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01e1747f-2e0e-47e7-b624-70dfcf1efe69 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:48:08.137463 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-01e1747f-2e0e-47e7-b624-70dfcf1efe69 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:48:08.248102 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:48:08.330577 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01e1747f-2e0e-47e7-b624-70dfcf1efe69 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:48:08.483237 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-310b30cc-545e-428f-97e9-26652325230d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '168b07df50c24593ba73dbde5b037aed', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:48:08.483891 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-310b30cc-545e-428f-97e9-26652325230d service nova] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:48:08.484168 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-310b30cc-545e-428f-97e9-26652325230d service nova] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:48:08.484434 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-310b30cc-545e-428f-97e9-26652325230d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:48:08.486235 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-310b30cc-545e-428f-97e9-26652325230d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:48:08.544755 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:48:08.623771 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-310b30cc-545e-428f-97e9-26652325230d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:48:08.625530 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-310b30cc-545e-428f-97e9-26652325230d service nova] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:48:08.648206 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 700/1388] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:48:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 513 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:48:10.605709 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44099f5b-4f4b-41b9-8011-99b009bfda50 None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:48:10.606584 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44099f5b-4f4b-41b9-8011-99b009bfda50 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:48:10.606584 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44099f5b-4f4b-41b9-8011-99b009bfda50 None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:48:10.608128 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 689/1389] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:48:10 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 28 00:48:10.615582 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc3a300d-2f0d-4351-b5cd-0faff27b110f None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:48:10.616169 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc3a300d-2f0d-4351-b5cd-0faff27b110f None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:48:10.616561 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc3a300d-2f0d-4351-b5cd-0faff27b110f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:48:10.711062 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-097bc6f9-ea7a-40af-b25f-6d05374bcd94 None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:48:10.711477 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-097bc6f9-ea7a-40af-b25f-6d05374bcd94 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:48:10.711920 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-097bc6f9-ea7a-40af-b25f-6d05374bcd94 None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:48:10.714119 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-fc3a300d-2f0d-4351-b5cd-0faff27b110f None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:48:10.714358 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 690/1390] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:48:10 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 28 00:48:10.715456 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-fc3a300d-2f0d-4351-b5cd-0faff27b110f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:48:10.720653 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d159df94-62c7-4f63-aeb5-a64c05e4ff09 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:48:10.721233 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d159df94-62c7-4f63-aeb5-a64c05e4ff09 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:48:10.721802 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d159df94-62c7-4f63-aeb5-a64c05e4ff09 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:48:10.772279 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:48:10.817323 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-d159df94-62c7-4f63-aeb5-a64c05e4ff09 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:48:10.818574 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-d159df94-62c7-4f63-aeb5-a64c05e4ff09 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:48:10.835028 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 701/1391] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:48:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 222 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:48:10.952209 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 691/1392] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:48:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 233 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 28 00:48:11.152844 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c399f32d-2d77-4b06-9ece-5ae435d59ad5 None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:48:11.153567 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c399f32d-2d77-4b06-9ece-5ae435d59ad5 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:48:11.153972 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c399f32d-2d77-4b06-9ece-5ae435d59ad5 None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:48:11.156671 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 702/1393] 10.222.0.42 () {54 vars in 967 bytes} [Sun Aug 28 00:48:11 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:48:11.162686 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a5dd123-8d7c-4a74-9b93-6f5f932deb5b None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:48:11.163085 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a5dd123-8d7c-4a74-9b93-6f5f932deb5b None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:48:11.163473 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a5dd123-8d7c-4a74-9b93-6f5f932deb5b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:48:11.266472 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-8a5dd123-8d7c-4a74-9b93-6f5f932deb5b None None] MFA Rules not processed for user `14176b54cf6640abb3b1c14e3824d4f1`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:48:11.268385 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-8a5dd123-8d7c-4a74-9b93-6f5f932deb5b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:48:11.286528 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c399f32d-2d77-4b06-9ece-5ae435d59ad5 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:48:11.288601 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-c399f32d-2d77-4b06-9ece-5ae435d59ad5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:48:11.311733 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:48:11.340727 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:48:11.382645 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 692/1394] 10.222.0.42 () {58 vars in 1064 bytes} [Sun Aug 28 00:48:11 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 221 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:48:11.390747 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a5dd123-8d7c-4a74-9b93-6f5f932deb5b None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:48:11.402185 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a5dd123-8d7c-4a74-9b93-6f5f932deb5b None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:48:11.510273 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c399f32d-2d77-4b06-9ece-5ae435d59ad5 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:48:11.514141 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93a17f93-dcc4-4286-a309-0be7ad51bccc service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14176b54cf6640abb3b1c14e3824d4f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:48:11.515199 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93a17f93-dcc4-4286-a309-0be7ad51bccc service glance] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:48:11.515686 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93a17f93-dcc4-4286-a309-0be7ad51bccc service glance] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:48:11.516334 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93a17f93-dcc4-4286-a309-0be7ad51bccc service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:48:11.519521 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-93a17f93-dcc4-4286-a309-0be7ad51bccc service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:48:11.626907 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:48:11.651841 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93a17f93-dcc4-4286-a309-0be7ad51bccc service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:48:11.653465 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93a17f93-dcc4-4286-a309-0be7ad51bccc service glance] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:48:11.660864 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85748a27-71a3-40ce-a92e-646b38104350 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '168b07df50c24593ba73dbde5b037aed', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:48:11.661522 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85748a27-71a3-40ce-a92e-646b38104350 service nova] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:48:11.661798 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85748a27-71a3-40ce-a92e-646b38104350 service nova] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:48:11.662066 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85748a27-71a3-40ce-a92e-646b38104350 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:48:11.663825 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-85748a27-71a3-40ce-a92e-646b38104350 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:48:11.675273 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:48:11.682224 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 693/1395] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:48:11 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 292 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 28 00:48:11.790239 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85748a27-71a3-40ce-a92e-646b38104350 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:48:11.791940 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85748a27-71a3-40ce-a92e-646b38104350 service nova] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:48:11.816641 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 703/1396] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:48:11 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 531 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 28 00:48:12.581829 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e66551e-0bc3-481d-a7a5-fce3e72e0a1e None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:48:12.582434 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e66551e-0bc3-481d-a7a5-fce3e72e0a1e None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:48:12.583127 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e66551e-0bc3-481d-a7a5-fce3e72e0a1e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:48:12.696235 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-5e66551e-0bc3-481d-a7a5-fce3e72e0a1e None None] MFA Rules not processed for user `eede08f020e04282bb994b855e371433`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:48:12.697894 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-5e66551e-0bc3-481d-a7a5-fce3e72e0a1e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:48:12.834961 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 694/1397] 10.222.0.42 () {58 vars in 1009 bytes} [Sun Aug 28 00:48:12 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 255 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:48:12.897390 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b788838c-0b43-4869-bc67-63845c4c96b4 None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:48:12.898212 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b788838c-0b43-4869-bc67-63845c4c96b4 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:48:12.898212 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b788838c-0b43-4869-bc67-63845c4c96b4 None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:48:12.899387 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 704/1398] 10.222.0.42 () {54 vars in 943 bytes} [Sun Aug 28 00:48:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:48:12.907881 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd81ae39-52b4-4ea8-90a9-f3f7d857418c None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:48:12.908545 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd81ae39-52b4-4ea8-90a9-f3f7d857418c None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:48:12.909408 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd81ae39-52b4-4ea8-90a9-f3f7d857418c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:48:13.057110 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-bd81ae39-52b4-4ea8-90a9-f3f7d857418c None None] MFA Rules not processed for user `ac4c5584582f4f55a7085af4cbbabf9b`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:48:13.058614 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-bd81ae39-52b4-4ea8-90a9-f3f7d857418c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:48:13.099487 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:48:13.249374 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 695/1399] 10.222.0.42 () {58 vars in 1040 bytes} [Sun Aug 28 00:48:12 2022] POST /identity/v3/auth/tokens => generated 3304 bytes in 344 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:48:13.262453 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b788838c-0b43-4869-bc67-63845c4c96b4 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:48:13.264216 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-b788838c-0b43-4869-bc67-63845c4c96b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:48:13.400936 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:48:13.462972 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b788838c-0b43-4869-bc67-63845c4c96b4 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:48:13.601940 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23d1c01d-46fd-4037-a31a-ed5d7fde4c35 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac4c5584582f4f55a7085af4cbbabf9b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:48:13.603008 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23d1c01d-46fd-4037-a31a-ed5d7fde4c35 service swift] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:48:13.603484 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23d1c01d-46fd-4037-a31a-ed5d7fde4c35 service swift] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:48:13.603962 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23d1c01d-46fd-4037-a31a-ed5d7fde4c35 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:48:13.608119 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-23d1c01d-46fd-4037-a31a-ed5d7fde4c35 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:48:13.726101 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23d1c01d-46fd-4037-a31a-ed5d7fde4c35 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:48:13.729007 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23d1c01d-46fd-4037-a31a-ed5d7fde4c35 service swift] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:48:13.734295 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:48:13.741637 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 705/1400] 10.222.0.42 () {60 vars in 1384 bytes} [Sun Aug 28 00:48:13 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 480 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 28 00:48:14.039611 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e209456-0a45-4843-a248-691a7e06f4c2 None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:48:14.040169 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e209456-0a45-4843-a248-691a7e06f4c2 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:48:14.040169 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e209456-0a45-4843-a248-691a7e06f4c2 None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:48:14.041863 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 696/1401] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:48:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:48:14.053835 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74c2ad60-d5e5-4395-8529-9062b0f2f524 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:48:14.054386 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74c2ad60-d5e5-4395-8529-9062b0f2f524 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:48:14.054803 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74c2ad60-d5e5-4395-8529-9062b0f2f524 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:48:14.184005 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-74c2ad60-d5e5-4395-8529-9062b0f2f524 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:48:14.185360 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-74c2ad60-d5e5-4395-8529-9062b0f2f524 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:48:14.342172 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 706/1402] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:48:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 290 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:48:14.734210 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e209456-0a45-4843-a248-691a7e06f4c2 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:48:14.736014 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-7e209456-0a45-4843-a248-691a7e06f4c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:48:14.943915 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e209456-0a45-4843-a248-691a7e06f4c2 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:48:14.947263 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:48:15.102819 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a4ab544-9e6d-4580-8650-74c40bd7cd2f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '168b07df50c24593ba73dbde5b037aed', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:48:15.103559 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a4ab544-9e6d-4580-8650-74c40bd7cd2f service nova] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:48:15.103921 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a4ab544-9e6d-4580-8650-74c40bd7cd2f service nova] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:48:15.104354 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a4ab544-9e6d-4580-8650-74c40bd7cd2f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:48:15.106314 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-5a4ab544-9e6d-4580-8650-74c40bd7cd2f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:48:15.216777 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a4ab544-9e6d-4580-8650-74c40bd7cd2f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:48:15.218495 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a4ab544-9e6d-4580-8650-74c40bd7cd2f service nova] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:48:15.251090 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 697/1403] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:48:14 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 513 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 28 00:48:17.707518 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2bb6a83-5cda-40c2-ac70-70b3cd2c3cfd None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:48:17.708713 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2bb6a83-5cda-40c2-ac70-70b3cd2c3cfd None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:48:17.708713 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2bb6a83-5cda-40c2-ac70-70b3cd2c3cfd None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:48:17.711046 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 707/1404] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:48:17 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:48:17.716926 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed5aa9d0-02f7-4393-a2aa-15494b59da79 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:48:17.717771 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed5aa9d0-02f7-4393-a2aa-15494b59da79 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:48:17.717771 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed5aa9d0-02f7-4393-a2aa-15494b59da79 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:48:17.732730 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:48:17.842334 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-ed5aa9d0-02f7-4393-a2aa-15494b59da79 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:48:17.845232 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-ed5aa9d0-02f7-4393-a2aa-15494b59da79 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:48:18.095113 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 698/1405] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:48:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 379 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 28 00:48:18.597436 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2bb6a83-5cda-40c2-ac70-70b3cd2c3cfd None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:48:18.600475 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-a2bb6a83-5cda-40c2-ac70-70b3cd2c3cfd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:48:18.775620 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:48:18.907702 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2bb6a83-5cda-40c2-ac70-70b3cd2c3cfd None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:48:19.083436 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74c6ec42-9754-4a65-a00c-98335723ac6a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '168b07df50c24593ba73dbde5b037aed', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:48:19.084178 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74c6ec42-9754-4a65-a00c-98335723ac6a service nova] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:48:19.084537 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74c6ec42-9754-4a65-a00c-98335723ac6a service nova] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:48:19.084876 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74c6ec42-9754-4a65-a00c-98335723ac6a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:48:19.086766 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-74c6ec42-9754-4a65-a00c-98335723ac6a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:48:19.216334 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:48:19.307623 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74c6ec42-9754-4a65-a00c-98335723ac6a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:48:19.309314 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74c6ec42-9754-4a65-a00c-98335723ac6a service nova] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:48:19.334666 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 708/1406] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:48:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 738 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:48:21.554432 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddd67739-e6ec-462f-acc3-18f9812bfcb3 None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:48:21.555761 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddd67739-e6ec-462f-acc3-18f9812bfcb3 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:48:21.556551 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddd67739-e6ec-462f-acc3-18f9812bfcb3 None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:48:21.565895 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 699/1407] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:48:21 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:48:21.567835 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d3d4c5-563e-4143-be37-b7b95b5d3644 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:48:21.568703 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d3d4c5-563e-4143-be37-b7b95b5d3644 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:48:21.569060 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d3d4c5-563e-4143-be37-b7b95b5d3644 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:48:21.678524 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-94d3d4c5-563e-4143-be37-b7b95b5d3644 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:48:21.679764 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-94d3d4c5-563e-4143-be37-b7b95b5d3644 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:48:21.820618 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 709/1408] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:48:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 254 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:48:22.116364 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddd67739-e6ec-462f-acc3-18f9812bfcb3 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:48:22.118062 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-ddd67739-e6ec-462f-acc3-18f9812bfcb3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:48:22.248379 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:48:22.346197 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddd67739-e6ec-462f-acc3-18f9812bfcb3 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:48:22.502774 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd75c392-55fc-47c4-969c-8057bf92cd4d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '168b07df50c24593ba73dbde5b037aed', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:48:22.503764 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd75c392-55fc-47c4-969c-8057bf92cd4d service nova] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:48:22.504214 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd75c392-55fc-47c4-969c-8057bf92cd4d service nova] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:48:22.504628 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd75c392-55fc-47c4-969c-8057bf92cd4d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:48:22.506459 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-fd75c392-55fc-47c4-969c-8057bf92cd4d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:48:22.581376 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:48:22.634677 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd75c392-55fc-47c4-969c-8057bf92cd4d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:48:22.636353 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd75c392-55fc-47c4-969c-8057bf92cd4d service nova] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:48:22.661893 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 700/1409] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:48:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 546 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:48:31.817096 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=93 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:48:32.658778 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=24 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:48:36.867552 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce95250f-2f80-4209-a74d-774942fcda42 None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:48:36.868140 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce95250f-2f80-4209-a74d-774942fcda42 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:48:36.868140 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce95250f-2f80-4209-a74d-774942fcda42 None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:48:36.869518 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 710/1410] 10.222.0.42 () {54 vars in 918 bytes} [Sun Aug 28 00:48:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:48:36.875308 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e379ce7a-a028-4ef6-8aee-4134655d3184 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:48:36.875814 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e379ce7a-a028-4ef6-8aee-4134655d3184 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:48:36.875914 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e379ce7a-a028-4ef6-8aee-4134655d3184 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:48:36.960672 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-e379ce7a-a028-4ef6-8aee-4134655d3184 None None] MFA Rules not processed for user `c2a3834fa6894dada3291e4781bbd553`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:48:36.961853 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-e379ce7a-a028-4ef6-8aee-4134655d3184 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:48:37.087963 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 701/1411] 10.222.0.42 () {58 vars in 1015 bytes} [Sun Aug 28 00:48:36 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 214 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:48:37.110728 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dae82f90-1e7d-40c2-86c1-1ff661102440 None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:48:37.111064 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dae82f90-1e7d-40c2-86c1-1ff661102440 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:48:37.111471 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dae82f90-1e7d-40c2-86c1-1ff661102440 None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:48:37.113276 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 711/1412] 10.222.0.42 () {54 vars in 918 bytes} [Sun Aug 28 00:48:37 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:48:37.118928 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c06491c5-63fb-4033-8cb2-02de591c2ee2 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:48:37.119244 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c06491c5-63fb-4033-8cb2-02de591c2ee2 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:48:37.119478 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c06491c5-63fb-4033-8cb2-02de591c2ee2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:48:37.202377 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:48:37.206204 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-c06491c5-63fb-4033-8cb2-02de591c2ee2 None None] MFA Rules not processed for user `168b07df50c24593ba73dbde5b037aed`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:48:37.208159 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-c06491c5-63fb-4033-8cb2-02de591c2ee2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:48:37.354427 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 702/1413] 10.222.0.42 () {58 vars in 1015 bytes} [Sun Aug 28 00:48:37 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 236 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:48:39.797008 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f065db-dd32-4468-9222-497727dcaf80 None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:48:39.797866 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f065db-dd32-4468-9222-497727dcaf80 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:48:39.797866 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f065db-dd32-4468-9222-497727dcaf80 None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:48:39.799595 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 712/1414] 127.0.0.1 () {54 vars in 917 bytes} [Sun Aug 28 00:48:39 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:48:39.806142 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62110057-97a5-4a7a-9533-39105ba34cec None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:48:39.807001 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62110057-97a5-4a7a-9533-39105ba34cec None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:48:39.807001 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62110057-97a5-4a7a-9533-39105ba34cec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:48:39.881560 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-62110057-97a5-4a7a-9533-39105ba34cec None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:48:39.882794 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-62110057-97a5-4a7a-9533-39105ba34cec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:48:39.943729 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:48:39.988216 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 703/1415] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Aug 28 00:48:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:48:40.288863 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6f065db-dd32-4468-9222-497727dcaf80 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:48:40.290743 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-f6f065db-dd32-4468-9222-497727dcaf80 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:48:40.447576 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6f065db-dd32-4468-9222-497727dcaf80 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:48:40.529328 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:48:40.573587 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cd0045f-b751-49be-92f2-105058a9d4c2 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '168b07df50c24593ba73dbde5b037aed', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:48:40.574192 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cd0045f-b751-49be-92f2-105058a9d4c2 service nova] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:48:40.574773 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cd0045f-b751-49be-92f2-105058a9d4c2 service nova] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:48:40.575040 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cd0045f-b751-49be-92f2-105058a9d4c2 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:48:40.576666 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-2cd0045f-b751-49be-92f2-105058a9d4c2 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:48:40.675467 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cd0045f-b751-49be-92f2-105058a9d4c2 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:48:40.677138 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cd0045f-b751-49be-92f2-105058a9d4c2 service nova] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:48:40.698481 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 713/1416] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:48:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 410 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:48:41.591904 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ba722e8-8009-402b-81d8-b023aed5c369 None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:48:41.592510 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ba722e8-8009-402b-81d8-b023aed5c369 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:48:41.592510 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ba722e8-8009-402b-81d8-b023aed5c369 None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:48:41.593937 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 704/1417] 127.0.0.1 () {54 vars in 901 bytes} [Sun Aug 28 00:48:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:48:41.599467 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db210022-5a8b-4397-a9f3-d6d44a0a8f06 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:48:41.599893 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db210022-5a8b-4397-a9f3-d6d44a0a8f06 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:48:41.599998 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db210022-5a8b-4397-a9f3-d6d44a0a8f06 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:48:41.684393 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-db210022-5a8b-4397-a9f3-d6d44a0a8f06 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:48:41.685686 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-db210022-5a8b-4397-a9f3-d6d44a0a8f06 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:48:41.714650 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:48:41.789330 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 714/1418] 127.0.0.1 () {58 vars in 998 bytes} [Sun Aug 28 00:48:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 191 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:48:41.937723 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ba722e8-8009-402b-81d8-b023aed5c369 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:48:41.939363 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-9ba722e8-8009-402b-81d8-b023aed5c369 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:48:42.094150 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ba722e8-8009-402b-81d8-b023aed5c369 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:48:42.137157 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:48:42.222706 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3387bb77-884b-4e4d-afb9-7b38872cdbac service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '168b07df50c24593ba73dbde5b037aed', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:48:42.223406 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3387bb77-884b-4e4d-afb9-7b38872cdbac service nova] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:48:42.223711 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3387bb77-884b-4e4d-afb9-7b38872cdbac service nova] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:48:42.224053 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3387bb77-884b-4e4d-afb9-7b38872cdbac service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:48:42.226167 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-3387bb77-884b-4e4d-afb9-7b38872cdbac service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:48:42.320988 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3387bb77-884b-4e4d-afb9-7b38872cdbac service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:48:42.322592 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3387bb77-884b-4e4d-afb9-7b38872cdbac service nova] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:48:42.341943 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 705/1419] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:48:41 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 405 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:48:43.241483 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2108c20a-03f9-42be-b582-421c6a1b648a None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:48:43.242207 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2108c20a-03f9-42be-b582-421c6a1b648a None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:48:43.242207 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2108c20a-03f9-42be-b582-421c6a1b648a None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:48:43.243769 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 715/1420] 127.0.0.1 () {54 vars in 901 bytes} [Sun Aug 28 00:48:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:48:43.249266 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39cbf3b9-1f84-4854-b445-afe2e1a72fbc None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:48:43.249771 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39cbf3b9-1f84-4854-b445-afe2e1a72fbc None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:48:43.249886 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39cbf3b9-1f84-4854-b445-afe2e1a72fbc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:48:43.315942 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:48:43.340129 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-39cbf3b9-1f84-4854-b445-afe2e1a72fbc None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:48:43.341444 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-39cbf3b9-1f84-4854-b445-afe2e1a72fbc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:48:43.447431 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 706/1421] 127.0.0.1 () {58 vars in 998 bytes} [Sun Aug 28 00:48:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 199 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:48:43.597008 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2108c20a-03f9-42be-b582-421c6a1b648a None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:48:43.602221 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-2108c20a-03f9-42be-b582-421c6a1b648a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:48:43.755692 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2108c20a-03f9-42be-b582-421c6a1b648a None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:48:43.768451 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:48:43.873261 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ddaa5ca-4ece-4b02-b6a8-4c3480ecd6a8 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '168b07df50c24593ba73dbde5b037aed', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:48:43.873875 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ddaa5ca-4ece-4b02-b6a8-4c3480ecd6a8 service nova] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:48:43.874149 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ddaa5ca-4ece-4b02-b6a8-4c3480ecd6a8 service nova] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:48:43.874411 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ddaa5ca-4ece-4b02-b6a8-4c3480ecd6a8 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:48:43.876079 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-6ddaa5ca-4ece-4b02-b6a8-4c3480ecd6a8 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:48:43.977302 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ddaa5ca-4ece-4b02-b6a8-4c3480ecd6a8 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:48:43.978921 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ddaa5ca-4ece-4b02-b6a8-4c3480ecd6a8 service nova] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:48:44.001137 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 716/1422] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:48:43 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 405 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:48:44.870451 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c9e5076-ddd0-461c-9bfd-ac91839316e9 None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:48:44.871159 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c9e5076-ddd0-461c-9bfd-ac91839316e9 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:48:44.871159 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c9e5076-ddd0-461c-9bfd-ac91839316e9 None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:48:44.872717 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 707/1423] 127.0.0.1 () {54 vars in 901 bytes} [Sun Aug 28 00:48:44 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:48:44.877936 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bb42965-7ca4-4864-a9d4-6ac6f4bf66a6 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:48:44.878430 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bb42965-7ca4-4864-a9d4-6ac6f4bf66a6 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:48:44.878430 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bb42965-7ca4-4864-a9d4-6ac6f4bf66a6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:48:44.899663 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:48:44.959476 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-5bb42965-7ca4-4864-a9d4-6ac6f4bf66a6 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:48:44.960590 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-5bb42965-7ca4-4864-a9d4-6ac6f4bf66a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:48:45.061716 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 717/1424] 127.0.0.1 () {58 vars in 998 bytes} [Sun Aug 28 00:48:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:48:45.215871 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c9e5076-ddd0-461c-9bfd-ac91839316e9 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:48:45.217362 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-2c9e5076-ddd0-461c-9bfd-ac91839316e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:48:45.351338 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:48:45.389012 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c9e5076-ddd0-461c-9bfd-ac91839316e9 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:48:45.507722 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34b35837-85f1-4a46-ad48-9a71bc780e10 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '168b07df50c24593ba73dbde5b037aed', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:48:45.508224 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34b35837-85f1-4a46-ad48-9a71bc780e10 service nova] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:48:45.508504 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34b35837-85f1-4a46-ad48-9a71bc780e10 service nova] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:48:45.508768 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34b35837-85f1-4a46-ad48-9a71bc780e10 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:48:45.510420 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-34b35837-85f1-4a46-ad48-9a71bc780e10 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:48:45.593215 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:48:45.608611 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34b35837-85f1-4a46-ad48-9a71bc780e10 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:48:45.610262 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34b35837-85f1-4a46-ad48-9a71bc780e10 service nova] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:48:45.628786 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 708/1425] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:48:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 413 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 28 00:48:46.498963 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d6c785b-1ba6-4530-9990-718be1c0a85b None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:48:46.499579 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d6c785b-1ba6-4530-9990-718be1c0a85b None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:48:46.499579 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d6c785b-1ba6-4530-9990-718be1c0a85b None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:48:46.500891 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 718/1426] 127.0.0.1 () {54 vars in 901 bytes} [Sun Aug 28 00:48:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:48:46.506125 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-442afd86-23a8-4ddb-90c5-3bbe262ca260 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:48:46.506608 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-442afd86-23a8-4ddb-90c5-3bbe262ca260 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:48:46.506986 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-442afd86-23a8-4ddb-90c5-3bbe262ca260 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:48:46.585746 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-442afd86-23a8-4ddb-90c5-3bbe262ca260 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:48:46.587007 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-442afd86-23a8-4ddb-90c5-3bbe262ca260 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:48:46.689619 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 709/1427] 127.0.0.1 () {58 vars in 998 bytes} [Sun Aug 28 00:48:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:48:46.840925 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d6c785b-1ba6-4530-9990-718be1c0a85b None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:48:46.842399 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-0d6c785b-1ba6-4530-9990-718be1c0a85b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:48:46.932663 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:48:47.001644 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d6c785b-1ba6-4530-9990-718be1c0a85b None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:48:47.120369 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d49ba017-c8df-462e-b4e5-7ad6fe97048d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '168b07df50c24593ba73dbde5b037aed', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:48:47.120997 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d49ba017-c8df-462e-b4e5-7ad6fe97048d service nova] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:48:47.121282 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d49ba017-c8df-462e-b4e5-7ad6fe97048d service nova] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:48:47.121588 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d49ba017-c8df-462e-b4e5-7ad6fe97048d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:48:47.123137 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-d49ba017-c8df-462e-b4e5-7ad6fe97048d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:48:47.171632 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:48:47.216948 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d49ba017-c8df-462e-b4e5-7ad6fe97048d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:48:47.218774 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d49ba017-c8df-462e-b4e5-7ad6fe97048d service nova] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:48:47.239383 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 719/1428] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:48:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 399 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:48:48.053946 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-200dbca7-7233-4ffd-89fd-3d034208f79c None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:48:48.054594 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-200dbca7-7233-4ffd-89fd-3d034208f79c None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:48:48.054594 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-200dbca7-7233-4ffd-89fd-3d034208f79c None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:48:48.055959 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 710/1429] 127.0.0.1 () {54 vars in 901 bytes} [Sun Aug 28 00:48:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:48:48.061413 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-010d257b-9f30-4651-a631-83b83a4d874d None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:48:48.061836 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-010d257b-9f30-4651-a631-83b83a4d874d None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:48:48.061936 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-010d257b-9f30-4651-a631-83b83a4d874d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:48:48.142479 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-010d257b-9f30-4651-a631-83b83a4d874d None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:48:48.143746 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-010d257b-9f30-4651-a631-83b83a4d874d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:48:48.243980 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 720/1430] 127.0.0.1 () {58 vars in 998 bytes} [Sun Aug 28 00:48:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 28 00:48:48.396888 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-200dbca7-7233-4ffd-89fd-3d034208f79c None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:48:48.398439 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-200dbca7-7233-4ffd-89fd-3d034208f79c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:48:48.448489 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:48:48.550816 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-200dbca7-7233-4ffd-89fd-3d034208f79c None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:48:48.669138 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7910aa38-b9b1-4979-8c0d-eeeac16cf8c2 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '168b07df50c24593ba73dbde5b037aed', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:48:48.669742 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7910aa38-b9b1-4979-8c0d-eeeac16cf8c2 service nova] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:48:48.670012 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7910aa38-b9b1-4979-8c0d-eeeac16cf8c2 service nova] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:48:48.670277 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7910aa38-b9b1-4979-8c0d-eeeac16cf8c2 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:48:48.671871 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-7910aa38-b9b1-4979-8c0d-eeeac16cf8c2 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:48:48.679101 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:48:48.768380 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7910aa38-b9b1-4979-8c0d-eeeac16cf8c2 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:48:48.770038 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7910aa38-b9b1-4979-8c0d-eeeac16cf8c2 service nova] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:48:48.789705 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 711/1431] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:48:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 393 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:48:49.615412 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c05c3f-31cd-4c73-aa56-0869c32d80e4 None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:48:49.616028 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c05c3f-31cd-4c73-aa56-0869c32d80e4 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:48:49.616028 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c05c3f-31cd-4c73-aa56-0869c32d80e4 None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:48:49.617416 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 721/1432] 127.0.0.1 () {54 vars in 901 bytes} [Sun Aug 28 00:48:49 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:48:49.622727 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cf40600-078e-412c-bf9a-c155e8e39b9b None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:48:49.623241 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cf40600-078e-412c-bf9a-c155e8e39b9b None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:48:49.623241 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cf40600-078e-412c-bf9a-c155e8e39b9b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:48:49.702199 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-8cf40600-078e-412c-bf9a-c155e8e39b9b None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:48:49.704171 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-8cf40600-078e-412c-bf9a-c155e8e39b9b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:48:49.789616 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:48:49.821771 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 712/1433] 127.0.0.1 () {58 vars in 998 bytes} [Sun Aug 28 00:48:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 200 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:48:49.970396 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65c05c3f-31cd-4c73-aa56-0869c32d80e4 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:48:49.971827 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-65c05c3f-31cd-4c73-aa56-0869c32d80e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:48:49.991457 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:48:50.128887 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65c05c3f-31cd-4c73-aa56-0869c32d80e4 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:48:50.231082 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:48:50.261173 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3181a5df-268d-47e2-b977-d82ab5561ab6 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '168b07df50c24593ba73dbde5b037aed', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:48:50.261788 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3181a5df-268d-47e2-b977-d82ab5561ab6 service nova] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:48:50.262111 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3181a5df-268d-47e2-b977-d82ab5561ab6 service nova] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:48:50.262386 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3181a5df-268d-47e2-b977-d82ab5561ab6 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:48:50.264058 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-3181a5df-268d-47e2-b977-d82ab5561ab6 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:48:50.373144 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3181a5df-268d-47e2-b977-d82ab5561ab6 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:48:50.374784 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3181a5df-268d-47e2-b977-d82ab5561ab6 service nova] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:48:50.395806 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 722/1434] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:48:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 426 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:48:51.211413 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cce8970-e836-43fc-a0c3-e3513ff3a528 None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:48:51.212005 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cce8970-e836-43fc-a0c3-e3513ff3a528 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:48:51.212005 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cce8970-e836-43fc-a0c3-e3513ff3a528 None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:48:51.213346 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 713/1435] 127.0.0.1 () {54 vars in 901 bytes} [Sun Aug 28 00:48:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:48:51.218733 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2f716d6-8b94-4a2d-befc-03dcc17a1ca6 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:48:51.219204 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2f716d6-8b94-4a2d-befc-03dcc17a1ca6 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:48:51.219305 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2f716d6-8b94-4a2d-befc-03dcc17a1ca6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:48:51.301014 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-b2f716d6-8b94-4a2d-befc-03dcc17a1ca6 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:48:51.302175 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-b2f716d6-8b94-4a2d-befc-03dcc17a1ca6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:48:51.348215 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:48:51.407451 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 723/1436] 127.0.0.1 () {58 vars in 998 bytes} [Sun Aug 28 00:48:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 190 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:48:51.559062 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cce8970-e836-43fc-a0c3-e3513ff3a528 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:48:51.560528 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-4cce8970-e836-43fc-a0c3-e3513ff3a528 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:48:51.725048 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cce8970-e836-43fc-a0c3-e3513ff3a528 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:48:51.781454 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:48:51.844398 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77dbf916-4993-4e27-ad76-3fdbb92bcdac service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '168b07df50c24593ba73dbde5b037aed', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:48:51.844934 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77dbf916-4993-4e27-ad76-3fdbb92bcdac service nova] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:48:51.845241 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77dbf916-4993-4e27-ad76-3fdbb92bcdac service nova] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:48:51.845508 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77dbf916-4993-4e27-ad76-3fdbb92bcdac service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:48:51.847150 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-77dbf916-4993-4e27-ad76-3fdbb92bcdac service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:48:51.941460 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77dbf916-4993-4e27-ad76-3fdbb92bcdac service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:48:51.943075 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77dbf916-4993-4e27-ad76-3fdbb92bcdac service nova] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:48:51.961595 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 714/1437] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:48:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 403 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:48:52.757621 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7d88896-b0e2-4622-b9f4-667b446eac1f None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:48:52.758184 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7d88896-b0e2-4622-b9f4-667b446eac1f None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:48:52.758184 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7d88896-b0e2-4622-b9f4-667b446eac1f None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:48:52.759496 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 724/1438] 127.0.0.1 () {54 vars in 901 bytes} [Sun Aug 28 00:48:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:48:52.765113 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-015fd38a-adce-4f90-aa1e-b28a87192c53 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:48:52.765583 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-015fd38a-adce-4f90-aa1e-b28a87192c53 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:48:52.765688 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-015fd38a-adce-4f90-aa1e-b28a87192c53 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:48:52.864005 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-015fd38a-adce-4f90-aa1e-b28a87192c53 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:48:52.865275 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-015fd38a-adce-4f90-aa1e-b28a87192c53 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:48:52.867438 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:48:52.976732 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 715/1439] 127.0.0.1 () {58 vars in 998 bytes} [Sun Aug 28 00:48:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 213 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:48:53.126965 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7d88896-b0e2-4622-b9f4-667b446eac1f None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:48:53.128825 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-a7d88896-b0e2-4622-b9f4-667b446eac1f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:48:53.307980 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7d88896-b0e2-4622-b9f4-667b446eac1f None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:48:53.336903 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:48:53.430793 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-725593e8-9a70-405d-b7cd-d179f0507b11 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '168b07df50c24593ba73dbde5b037aed', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:48:53.431409 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-725593e8-9a70-405d-b7cd-d179f0507b11 service nova] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:48:53.431677 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-725593e8-9a70-405d-b7cd-d179f0507b11 service nova] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:48:53.431943 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-725593e8-9a70-405d-b7cd-d179f0507b11 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:48:53.433622 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-725593e8-9a70-405d-b7cd-d179f0507b11 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:48:53.531699 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-725593e8-9a70-405d-b7cd-d179f0507b11 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:48:53.533348 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-725593e8-9a70-405d-b7cd-d179f0507b11 service nova] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:48:53.552286 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 725/1440] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:48:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 426 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:48:55.348911 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77e5b48c-2d6b-47c6-b753-910428572c41 None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:48:55.349530 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77e5b48c-2d6b-47c6-b753-910428572c41 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:48:55.349530 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77e5b48c-2d6b-47c6-b753-910428572c41 None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:48:55.350855 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 716/1441] 127.0.0.1 () {54 vars in 917 bytes} [Sun Aug 28 00:48:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:48:55.356333 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-809c2277-d58f-4675-b769-0f8453fce3aa None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:48:55.356792 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-809c2277-d58f-4675-b769-0f8453fce3aa None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:48:55.356901 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-809c2277-d58f-4675-b769-0f8453fce3aa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:48:55.399816 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:48:55.427374 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-809c2277-d58f-4675-b769-0f8453fce3aa None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:48:55.429328 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-809c2277-d58f-4675-b769-0f8453fce3aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:48:55.542339 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 726/1442] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Aug 28 00:48:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:48:55.820993 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77e5b48c-2d6b-47c6-b753-910428572c41 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:48:55.822720 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-77e5b48c-2d6b-47c6-b753-910428572c41 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:48:55.942269 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77e5b48c-2d6b-47c6-b753-910428572c41 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:48:55.979426 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:48:56.029843 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-254806ce-6456-491e-9596-fca31c69a505 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:48:56.030481 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-254806ce-6456-491e-9596-fca31c69a505 service neutron] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:48:56.030755 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-254806ce-6456-491e-9596-fca31c69a505 service neutron] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:48:56.031023 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-254806ce-6456-491e-9596-fca31c69a505 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:48:56.032785 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-254806ce-6456-491e-9596-fca31c69a505 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:48:56.156518 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-254806ce-6456-491e-9596-fca31c69a505 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:48:56.158019 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-254806ce-6456-491e-9596-fca31c69a505 service neutron] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:48:56.177564 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 717/1443] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:48:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 357 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:48:58.323827 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc96acb8-3a79-4559-b1af-d0b9168ecb69 None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:48:58.324390 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc96acb8-3a79-4559-b1af-d0b9168ecb69 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:48:58.324390 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc96acb8-3a79-4559-b1af-d0b9168ecb69 None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:48:58.326017 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 727/1444] 127.0.0.1 () {54 vars in 917 bytes} [Sun Aug 28 00:48:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:48:58.332001 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2559d2ac-1d83-458d-96c6-0ccd92581636 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:48:58.332538 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2559d2ac-1d83-458d-96c6-0ccd92581636 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:48:58.332538 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2559d2ac-1d83-458d-96c6-0ccd92581636 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:48:58.402586 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-2559d2ac-1d83-458d-96c6-0ccd92581636 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:48:58.403816 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-2559d2ac-1d83-458d-96c6-0ccd92581636 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:48:58.418059 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:48:58.508654 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 718/1445] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Aug 28 00:48:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:48:58.799401 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc96acb8-3a79-4559-b1af-d0b9168ecb69 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:48:58.800947 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-bc96acb8-3a79-4559-b1af-d0b9168ecb69 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:48:58.912706 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc96acb8-3a79-4559-b1af-d0b9168ecb69 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:48:58.926618 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:48:58.992076 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-864230d8-ad35-4e2d-a5f2-a24f1a290355 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:48:58.992564 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-864230d8-ad35-4e2d-a5f2-a24f1a290355 service neutron] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:48:58.992868 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-864230d8-ad35-4e2d-a5f2-a24f1a290355 service neutron] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:48:58.993165 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-864230d8-ad35-4e2d-a5f2-a24f1a290355 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:48:58.994807 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-864230d8-ad35-4e2d-a5f2-a24f1a290355 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:48:59.114308 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-864230d8-ad35-4e2d-a5f2-a24f1a290355 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:48:59.116951 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-864230d8-ad35-4e2d-a5f2-a24f1a290355 service neutron] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:48:59.138162 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 728/1446] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:48:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 339 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 28 00:49:01.433771 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11e0b70a-aa44-4f5d-8001-c7fec4cb8bd2 None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:49:01.434629 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11e0b70a-aa44-4f5d-8001-c7fec4cb8bd2 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:49:01.434629 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11e0b70a-aa44-4f5d-8001-c7fec4cb8bd2 None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:49:01.436041 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 719/1447] 127.0.0.1 () {54 vars in 917 bytes} [Sun Aug 28 00:49:01 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:49:01.442721 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31abf4ef-9432-4df1-9031-33b6d1b70901 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:49:01.443530 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31abf4ef-9432-4df1-9031-33b6d1b70901 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:49:01.443724 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31abf4ef-9432-4df1-9031-33b6d1b70901 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:49:01.501059 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:49:01.522174 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-31abf4ef-9432-4df1-9031-33b6d1b70901 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:49:01.522174 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-31abf4ef-9432-4df1-9031-33b6d1b70901 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:49:01.654860 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 729/1448] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Aug 28 00:49:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 213 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:49:01.938419 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11e0b70a-aa44-4f5d-8001-c7fec4cb8bd2 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:49:01.940014 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-11e0b70a-aa44-4f5d-8001-c7fec4cb8bd2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:49:02.050748 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11e0b70a-aa44-4f5d-8001-c7fec4cb8bd2 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:49:02.098543 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:49:02.140312 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36ec7a44-f9b4-4e29-9f61-e6562df1605f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:49:02.141021 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36ec7a44-f9b4-4e29-9f61-e6562df1605f service neutron] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:49:02.141303 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36ec7a44-f9b4-4e29-9f61-e6562df1605f service neutron] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:49:02.141577 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36ec7a44-f9b4-4e29-9f61-e6562df1605f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:49:02.143404 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-36ec7a44-f9b4-4e29-9f61-e6562df1605f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:49:02.268284 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36ec7a44-f9b4-4e29-9f61-e6562df1605f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:49:02.269727 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36ec7a44-f9b4-4e29-9f61-e6562df1605f service neutron] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:49:02.293262 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 720/1449] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:49:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 355 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 28 00:49:04.482893 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba64d972-4613-4a57-bf6d-1222f7b9bd00 None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:49:04.483530 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba64d972-4613-4a57-bf6d-1222f7b9bd00 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:49:04.483530 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba64d972-4613-4a57-bf6d-1222f7b9bd00 None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:49:04.484841 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 730/1450] 127.0.0.1 () {54 vars in 917 bytes} [Sun Aug 28 00:49:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:49:04.491410 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d96d9f05-c32e-4079-bcba-ba229661ff0b None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:49:04.492038 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d96d9f05-c32e-4079-bcba-ba229661ff0b None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:49:04.492038 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d96d9f05-c32e-4079-bcba-ba229661ff0b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:49:04.572717 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-d96d9f05-c32e-4079-bcba-ba229661ff0b None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:49:04.574033 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-d96d9f05-c32e-4079-bcba-ba229661ff0b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:49:04.599789 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:49:04.685106 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 721/1451] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Aug 28 00:49:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 195 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:49:04.971493 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba64d972-4613-4a57-bf6d-1222f7b9bd00 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:49:04.974734 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-ba64d972-4613-4a57-bf6d-1222f7b9bd00 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:49:05.106141 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba64d972-4613-4a57-bf6d-1222f7b9bd00 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:49:05.130187 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:49:05.198487 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11d26645-c8e5-4e2b-b9db-c0eea09bb216 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:49:05.199401 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11d26645-c8e5-4e2b-b9db-c0eea09bb216 service neutron] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:49:05.199700 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11d26645-c8e5-4e2b-b9db-c0eea09bb216 service neutron] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:49:05.200443 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11d26645-c8e5-4e2b-b9db-c0eea09bb216 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:49:05.203024 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-11d26645-c8e5-4e2b-b9db-c0eea09bb216 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:49:05.336073 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11d26645-c8e5-4e2b-b9db-c0eea09bb216 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:49:05.337577 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11d26645-c8e5-4e2b-b9db-c0eea09bb216 service neutron] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:49:05.358563 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 731/1452] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:49:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 388 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:49:07.395678 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77c138fc-5d46-43cc-bb23-8689ecc9f895 None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:49:07.396231 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77c138fc-5d46-43cc-bb23-8689ecc9f895 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:49:07.396350 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77c138fc-5d46-43cc-bb23-8689ecc9f895 None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:49:07.397908 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 722/1453] 127.0.0.1 () {54 vars in 917 bytes} [Sun Aug 28 00:49:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:49:07.403819 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70456752-06e8-4b4e-88c2-5172db546b67 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:49:07.404368 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70456752-06e8-4b4e-88c2-5172db546b67 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:49:07.404368 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70456752-06e8-4b4e-88c2-5172db546b67 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:49:07.474005 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:49:07.477356 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-70456752-06e8-4b4e-88c2-5172db546b67 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:49:07.479367 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-70456752-06e8-4b4e-88c2-5172db546b67 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:49:07.591182 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 732/1454] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Aug 28 00:49:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:49:07.890263 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77c138fc-5d46-43cc-bb23-8689ecc9f895 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:49:07.892094 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-77c138fc-5d46-43cc-bb23-8689ecc9f895 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:49:08.016068 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77c138fc-5d46-43cc-bb23-8689ecc9f895 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:49:08.072005 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:49:08.098949 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90c1670a-ac41-4228-b559-9c6393ac2562 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:49:08.099564 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90c1670a-ac41-4228-b559-9c6393ac2562 service neutron] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:49:08.099863 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90c1670a-ac41-4228-b559-9c6393ac2562 service neutron] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:49:08.100123 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90c1670a-ac41-4228-b559-9c6393ac2562 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:49:08.101890 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-90c1670a-ac41-4228-b559-9c6393ac2562 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:49:08.223501 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90c1670a-ac41-4228-b559-9c6393ac2562 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:49:08.224888 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90c1670a-ac41-4228-b559-9c6393ac2562 service neutron] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:49:08.247291 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 723/1455] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:49:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 358 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:49:10.348635 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aef05e6-3888-4038-9a91-1235c8d3e7ae None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:49:10.349197 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aef05e6-3888-4038-9a91-1235c8d3e7ae None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:49:10.349197 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aef05e6-3888-4038-9a91-1235c8d3e7ae None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:49:10.350542 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 733/1456] 127.0.0.1 () {54 vars in 917 bytes} [Sun Aug 28 00:49:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:49:10.356151 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87cb4135-222c-426f-aeb8-4a5cdc408cc6 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:49:10.356585 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87cb4135-222c-426f-aeb8-4a5cdc408cc6 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:49:10.356696 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87cb4135-222c-426f-aeb8-4a5cdc408cc6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:49:10.423751 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-87cb4135-222c-426f-aeb8-4a5cdc408cc6 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:49:10.425001 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-87cb4135-222c-426f-aeb8-4a5cdc408cc6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:49:10.455593 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:49:10.528430 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 724/1457] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Aug 28 00:49:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:49:10.809442 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aef05e6-3888-4038-9a91-1235c8d3e7ae None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:49:10.811030 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-9aef05e6-3888-4038-9a91-1235c8d3e7ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:49:10.919716 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aef05e6-3888-4038-9a91-1235c8d3e7ae None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:49:10.950445 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:49:11.001702 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ef64938-1361-4aba-8be6-ed68d1e858d4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:49:11.002230 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef64938-1361-4aba-8be6-ed68d1e858d4 service neutron] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:49:11.002503 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef64938-1361-4aba-8be6-ed68d1e858d4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:49:11.002775 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef64938-1361-4aba-8be6-ed68d1e858d4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:49:11.004379 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-4ef64938-1361-4aba-8be6-ed68d1e858d4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:49:11.118100 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ef64938-1361-4aba-8be6-ed68d1e858d4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:49:11.119589 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ef64938-1361-4aba-8be6-ed68d1e858d4 service neutron] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:49:11.138868 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 734/1458] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:49:10 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 330 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:49:13.128950 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43be28ce-8586-480d-a156-1867bc3b3d90 None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:49:13.129499 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43be28ce-8586-480d-a156-1867bc3b3d90 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:49:13.129499 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43be28ce-8586-480d-a156-1867bc3b3d90 None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:49:13.130892 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 725/1459] 127.0.0.1 () {54 vars in 917 bytes} [Sun Aug 28 00:49:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:49:13.136550 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d032e1e4-9cdd-43e2-8076-3b69ab8d2dc0 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:49:13.137037 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d032e1e4-9cdd-43e2-8076-3b69ab8d2dc0 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:49:13.137147 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d032e1e4-9cdd-43e2-8076-3b69ab8d2dc0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:49:13.210286 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-d032e1e4-9cdd-43e2-8076-3b69ab8d2dc0 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:49:13.211623 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-d032e1e4-9cdd-43e2-8076-3b69ab8d2dc0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:49:13.221395 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:49:13.315243 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 735/1460] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Aug 28 00:49:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:49:13.596822 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43be28ce-8586-480d-a156-1867bc3b3d90 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:49:13.598448 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-43be28ce-8586-480d-a156-1867bc3b3d90 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:49:13.716769 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43be28ce-8586-480d-a156-1867bc3b3d90 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:49:13.775867 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:49:13.796035 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8ab6501-96b6-4320-be23-701e5413cb01 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:49:13.796649 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8ab6501-96b6-4320-be23-701e5413cb01 service neutron] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:49:13.797042 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8ab6501-96b6-4320-be23-701e5413cb01 service neutron] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:49:13.797284 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8ab6501-96b6-4320-be23-701e5413cb01 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:49:13.798917 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-a8ab6501-96b6-4320-be23-701e5413cb01 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:49:13.915568 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8ab6501-96b6-4320-be23-701e5413cb01 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:49:13.917055 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8ab6501-96b6-4320-be23-701e5413cb01 service neutron] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:49:13.937320 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 726/1461] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:49:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 341 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:49:15.914694 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3ed2ddf-0551-4cff-b33b-bb9eddd70c79 None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:49:15.915309 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3ed2ddf-0551-4cff-b33b-bb9eddd70c79 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:49:15.915309 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3ed2ddf-0551-4cff-b33b-bb9eddd70c79 None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:49:15.916566 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 736/1462] 127.0.0.1 () {54 vars in 917 bytes} [Sun Aug 28 00:49:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:49:15.922065 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa47a289-0a82-4747-95c4-461b42829327 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:49:15.922520 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa47a289-0a82-4747-95c4-461b42829327 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:49:15.922613 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa47a289-0a82-4747-95c4-461b42829327 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:49:15.995240 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-fa47a289-0a82-4747-95c4-461b42829327 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:49:15.996443 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-fa47a289-0a82-4747-95c4-461b42829327 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:49:16.034777 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:49:16.098121 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 727/1463] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Aug 28 00:49:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:49:16.388087 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3ed2ddf-0551-4cff-b33b-bb9eddd70c79 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:49:16.389693 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-c3ed2ddf-0551-4cff-b33b-bb9eddd70c79 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:49:16.496900 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3ed2ddf-0551-4cff-b33b-bb9eddd70c79 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:49:16.535953 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:49:16.573511 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b919549-a069-49d7-a784-70e213687957 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:49:16.574119 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b919549-a069-49d7-a784-70e213687957 service neutron] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:49:16.574366 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b919549-a069-49d7-a784-70e213687957 service neutron] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:49:16.574666 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b919549-a069-49d7-a784-70e213687957 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:49:16.576602 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-2b919549-a069-49d7-a784-70e213687957 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:49:16.688507 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b919549-a069-49d7-a784-70e213687957 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:49:16.689939 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b919549-a069-49d7-a784-70e213687957 service neutron] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:49:16.708921 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 737/1464] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:49:16 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 321 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:49:18.632020 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59809b6c-30e6-4553-8a66-e7ac6992270a None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:49:18.632602 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59809b6c-30e6-4553-8a66-e7ac6992270a None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:49:18.632841 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59809b6c-30e6-4553-8a66-e7ac6992270a None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:49:18.634727 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 728/1465] 127.0.0.1 () {54 vars in 917 bytes} [Sun Aug 28 00:49:18 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:49:18.640738 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85df7971-02c7-45bf-9d2a-4efb73e270f6 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:49:18.641386 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85df7971-02c7-45bf-9d2a-4efb73e270f6 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:49:18.641386 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85df7971-02c7-45bf-9d2a-4efb73e270f6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:49:18.711216 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-85df7971-02c7-45bf-9d2a-4efb73e270f6 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:49:18.712444 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-85df7971-02c7-45bf-9d2a-4efb73e270f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:49:18.731996 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:49:18.816007 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 738/1466] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Aug 28 00:49:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:49:19.097063 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59809b6c-30e6-4553-8a66-e7ac6992270a None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:49:19.099288 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-59809b6c-30e6-4553-8a66-e7ac6992270a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:49:19.240888 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59809b6c-30e6-4553-8a66-e7ac6992270a None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:49:19.310338 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:49:19.318727 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-481889e8-a97e-4afc-a567-bbe37e0cfb48 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:49:19.319349 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-481889e8-a97e-4afc-a567-bbe37e0cfb48 service neutron] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:49:19.319612 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-481889e8-a97e-4afc-a567-bbe37e0cfb48 service neutron] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:49:19.319865 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-481889e8-a97e-4afc-a567-bbe37e0cfb48 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:49:19.321593 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-481889e8-a97e-4afc-a567-bbe37e0cfb48 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:49:19.449296 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-481889e8-a97e-4afc-a567-bbe37e0cfb48 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:49:19.451058 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-481889e8-a97e-4afc-a567-bbe37e0cfb48 service neutron] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:49:19.471552 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 729/1467] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:49:19 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 375 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:49:21.421706 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e899232-df2b-4ac5-9ac6-0a9e71131c29 None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:49:21.422298 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e899232-df2b-4ac5-9ac6-0a9e71131c29 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:49:21.422298 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e899232-df2b-4ac5-9ac6-0a9e71131c29 None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:49:21.423629 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 739/1468] 127.0.0.1 () {54 vars in 917 bytes} [Sun Aug 28 00:49:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:49:21.429060 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72537e2b-d6bc-4eba-b5e8-eabfc02d74dc None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:49:21.429544 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72537e2b-d6bc-4eba-b5e8-eabfc02d74dc None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:49:21.429642 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72537e2b-d6bc-4eba-b5e8-eabfc02d74dc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:49:21.496912 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-72537e2b-d6bc-4eba-b5e8-eabfc02d74dc None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:49:21.498176 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-72537e2b-d6bc-4eba-b5e8-eabfc02d74dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:49:21.546725 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:49:21.595720 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 730/1469] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Aug 28 00:49:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:49:21.892457 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e899232-df2b-4ac5-9ac6-0a9e71131c29 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:49:21.894065 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-7e899232-df2b-4ac5-9ac6-0a9e71131c29 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:49:22.001232 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e899232-df2b-4ac5-9ac6-0a9e71131c29 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:49:22.057182 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:49:22.086925 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0083a853-8abc-4cab-92ca-c4ea132ab974 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:49:22.087546 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0083a853-8abc-4cab-92ca-c4ea132ab974 service neutron] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:49:22.087826 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0083a853-8abc-4cab-92ca-c4ea132ab974 service neutron] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:49:22.088092 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0083a853-8abc-4cab-92ca-c4ea132ab974 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:49:22.089789 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-0083a853-8abc-4cab-92ca-c4ea132ab974 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:49:22.208256 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0083a853-8abc-4cab-92ca-c4ea132ab974 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:49:22.209665 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0083a853-8abc-4cab-92ca-c4ea132ab974 service neutron] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:49:22.239531 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 740/1470] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:49:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 348 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:49:31.594023 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=22 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:49:32.236880 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=66 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:49:41.758932 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-430586e2-ca60-4109-8b0d-8fd0dd7e1658 None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:49:41.759497 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-430586e2-ca60-4109-8b0d-8fd0dd7e1658 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:49:41.759497 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-430586e2-ca60-4109-8b0d-8fd0dd7e1658 None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:49:41.760842 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 731/1471] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:49:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:49:41.766365 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-972d34f2-bd4b-4513-ad4e-43241b832848 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:49:41.766820 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-972d34f2-bd4b-4513-ad4e-43241b832848 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:49:41.766920 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-972d34f2-bd4b-4513-ad4e-43241b832848 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:49:41.836876 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-972d34f2-bd4b-4513-ad4e-43241b832848 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:49:41.838134 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-972d34f2-bd4b-4513-ad4e-43241b832848 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:49:41.941556 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 741/1472] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:49:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:49:42.238668 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a17c784-80e6-44f6-a862-ba6f7258543b None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:49:42.238951 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a17c784-80e6-44f6-a862-ba6f7258543b None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:49:42.239220 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a17c784-80e6-44f6-a862-ba6f7258543b None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:49:42.240632 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 732/1473] 10.222.0.42 () {54 vars in 967 bytes} [Sun Aug 28 00:49:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:49:42.245543 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6abb2be8-dcc4-4a2f-8747-2637b9ffe9a1 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:49:42.245828 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6abb2be8-dcc4-4a2f-8747-2637b9ffe9a1 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:49:42.246114 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6abb2be8-dcc4-4a2f-8747-2637b9ffe9a1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:49:42.325856 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-6abb2be8-dcc4-4a2f-8747-2637b9ffe9a1 None None] MFA Rules not processed for user `14176b54cf6640abb3b1c14e3824d4f1`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:49:42.327072 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-6abb2be8-dcc4-4a2f-8747-2637b9ffe9a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:49:42.348866 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:49:42.406447 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 742/1474] 10.222.0.42 () {58 vars in 1064 bytes} [Sun Aug 28 00:49:42 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:49:42.414407 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a17c784-80e6-44f6-a862-ba6f7258543b None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:49:42.415966 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-2a17c784-80e6-44f6-a862-ba6f7258543b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:49:42.525315 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a17c784-80e6-44f6-a862-ba6f7258543b None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:49:42.606502 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee15274c-253d-4d9d-9d9c-2b211345b565 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14176b54cf6640abb3b1c14e3824d4f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:49:42.607136 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee15274c-253d-4d9d-9d9c-2b211345b565 service glance] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:49:42.607377 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee15274c-253d-4d9d-9d9c-2b211345b565 service glance] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:49:42.607673 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee15274c-253d-4d9d-9d9c-2b211345b565 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:49:42.609512 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-ee15274c-253d-4d9d-9d9c-2b211345b565 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:49:42.674217 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:49:42.734449 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee15274c-253d-4d9d-9d9c-2b211345b565 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:49:42.735920 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee15274c-253d-4d9d-9d9c-2b211345b565 service glance] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:49:42.755514 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 733/1475] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:49:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 342 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:49:44.724470 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-027b17c7-ed7d-463e-985b-61ecc42e925b None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:49:44.725082 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-027b17c7-ed7d-463e-985b-61ecc42e925b None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:49:44.725082 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-027b17c7-ed7d-463e-985b-61ecc42e925b None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:49:44.727022 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 743/1476] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:49:44 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:49:44.732636 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b064cdb-98ab-4d26-a5e6-f24aa24dab12 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:49:44.733073 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b064cdb-98ab-4d26-a5e6-f24aa24dab12 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:49:44.733312 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b064cdb-98ab-4d26-a5e6-f24aa24dab12 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:49:44.807225 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-0b064cdb-98ab-4d26-a5e6-f24aa24dab12 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:49:44.808494 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-0b064cdb-98ab-4d26-a5e6-f24aa24dab12 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:49:44.931082 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 734/1477] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:49:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 199 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:49:45.199602 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-027b17c7-ed7d-463e-985b-61ecc42e925b None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:49:45.201515 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-027b17c7-ed7d-463e-985b-61ecc42e925b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:49:45.332352 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-027b17c7-ed7d-463e-985b-61ecc42e925b None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:49:45.397951 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:49:45.447561 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db50ed19-1759-4a49-9840-ae57e760008a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:49:45.448186 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db50ed19-1759-4a49-9840-ae57e760008a admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:49:45.448465 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db50ed19-1759-4a49-9840-ae57e760008a admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:49:45.448741 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db50ed19-1759-4a49-9840-ae57e760008a admin admin] PATH_INFO: `/v3/projects` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:49:45.449450 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db50ed19-1759-4a49-9840-ae57e760008a admin admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:49:45.451618 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: WARNING py.warnings [None req-db50ed19-1759-4a49-9840-ae57e760008a admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_projects": "(role:reader and system_scope:all) or (role:reader and domain_id:%(target.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 28 00:49:45.451618 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: warnings.warn(msg) Aug 28 00:49:45.451618 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: Aug 28 00:49:45.452820 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db50ed19-1759-4a49-9840-ae57e760008a admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:49:45.474192 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 744/1478] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Aug 28 00:49:45 2022] GET /identity/v3/projects => generated 2553 bytes in 275 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 00:49:47.396163 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d622f2d-3591-40d4-be83-751397a98581 None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:49:47.397132 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d622f2d-3591-40d4-be83-751397a98581 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:49:47.397132 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d622f2d-3591-40d4-be83-751397a98581 None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:49:47.398129 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 735/1479] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:49:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:49:47.404338 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7de189a-9d6f-4a23-b43a-4c82e277e0a3 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:49:47.405007 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7de189a-9d6f-4a23-b43a-4c82e277e0a3 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:49:47.405007 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7de189a-9d6f-4a23-b43a-4c82e277e0a3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:49:47.472621 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-b7de189a-9d6f-4a23-b43a-4c82e277e0a3 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:49:47.473849 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-b7de189a-9d6f-4a23-b43a-4c82e277e0a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:49:47.573846 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 745/1480] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:49:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:49:47.869870 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d622f2d-3591-40d4-be83-751397a98581 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:49:47.871444 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-3d622f2d-3591-40d4-be83-751397a98581 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:49:47.882008 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:49:48.030685 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d622f2d-3591-40d4-be83-751397a98581 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:49:48.120113 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:49:48.160578 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d3633b2-b752-449c-8d3d-eab9ca9dae88 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '168b07df50c24593ba73dbde5b037aed', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:49:48.161232 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d3633b2-b752-449c-8d3d-eab9ca9dae88 service nova] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:49:48.161517 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d3633b2-b752-449c-8d3d-eab9ca9dae88 service nova] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:49:48.161793 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d3633b2-b752-449c-8d3d-eab9ca9dae88 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:49:48.163453 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-6d3633b2-b752-449c-8d3d-eab9ca9dae88 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:49:48.264845 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d3633b2-b752-449c-8d3d-eab9ca9dae88 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:49:48.266451 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d3633b2-b752-449c-8d3d-eab9ca9dae88 service nova] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:49:48.287430 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 736/1481] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:49:47 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 418 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:49:50.382393 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33f044cb-aade-4448-8884-2e1b2e7bff85 None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:49:50.382964 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33f044cb-aade-4448-8884-2e1b2e7bff85 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:49:50.382964 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33f044cb-aade-4448-8884-2e1b2e7bff85 None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:49:50.384325 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 746/1482] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:49:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:49:50.390067 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff7f082-cbea-4433-8e18-ff9c067bf57a None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:49:50.390555 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff7f082-cbea-4433-8e18-ff9c067bf57a None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:49:50.390555 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff7f082-cbea-4433-8e18-ff9c067bf57a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:49:50.462209 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-4ff7f082-cbea-4433-8e18-ff9c067bf57a None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:49:50.463446 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-4ff7f082-cbea-4433-8e18-ff9c067bf57a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:49:50.511047 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:49:50.564000 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 737/1483] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:49:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:49:50.854020 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33f044cb-aade-4448-8884-2e1b2e7bff85 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:49:50.855580 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-33f044cb-aade-4448-8884-2e1b2e7bff85 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:49:50.888426 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:49:50.982810 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33f044cb-aade-4448-8884-2e1b2e7bff85 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:49:51.072930 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17bff16f-da42-4827-828d-48fc9fd38e13 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:49:51.073601 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17bff16f-da42-4827-828d-48fc9fd38e13 service neutron] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:49:51.073877 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17bff16f-da42-4827-828d-48fc9fd38e13 service neutron] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:49:51.074142 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17bff16f-da42-4827-828d-48fc9fd38e13 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:49:51.076437 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-17bff16f-da42-4827-828d-48fc9fd38e13 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:49:51.177213 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:49:51.220787 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17bff16f-da42-4827-828d-48fc9fd38e13 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:49:51.222360 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17bff16f-da42-4827-828d-48fc9fd38e13 service neutron] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:49:51.241250 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 747/1484] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:49:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 388 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:49:53.237683 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6b77aad-0799-4adb-a6d9-7ca4d0bd6153 None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:49:53.238283 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6b77aad-0799-4adb-a6d9-7ca4d0bd6153 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:49:53.238283 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6b77aad-0799-4adb-a6d9-7ca4d0bd6153 None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:49:53.239586 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 738/1485] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:49:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:49:53.245229 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d8a45a3-4b4f-4b87-a1c1-3892e0dd751a None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:49:53.245680 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d8a45a3-4b4f-4b87-a1c1-3892e0dd751a None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:49:53.245801 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d8a45a3-4b4f-4b87-a1c1-3892e0dd751a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:49:53.313328 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-5d8a45a3-4b4f-4b87-a1c1-3892e0dd751a None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:49:53.314913 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-5d8a45a3-4b4f-4b87-a1c1-3892e0dd751a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:49:53.416845 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 748/1486] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:49:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:49:53.708191 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6b77aad-0799-4adb-a6d9-7ca4d0bd6153 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:49:53.709814 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-b6b77aad-0799-4adb-a6d9-7ca4d0bd6153 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:49:53.825943 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6b77aad-0799-4adb-a6d9-7ca4d0bd6153 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:49:53.904882 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:49:53.907039 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-317435a4-269d-482d-83ae-a03c6630ecc3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:49:53.907837 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-317435a4-269d-482d-83ae-a03c6630ecc3 service neutron] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:49:53.908364 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-317435a4-269d-482d-83ae-a03c6630ecc3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:49:53.908796 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-317435a4-269d-482d-83ae-a03c6630ecc3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:49:53.910884 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-317435a4-269d-482d-83ae-a03c6630ecc3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:49:54.035847 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-317435a4-269d-482d-83ae-a03c6630ecc3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:49:54.037272 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-317435a4-269d-482d-83ae-a03c6630ecc3 service neutron] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:49:54.057446 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 739/1487] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:49:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 350 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:49:56.098288 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88eca543-fd98-4341-8643-5951b615ad08 None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:49:56.099014 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88eca543-fd98-4341-8643-5951b615ad08 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:49:56.099014 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88eca543-fd98-4341-8643-5951b615ad08 None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:49:56.100928 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 749/1488] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:49:56 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:49:56.107609 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58ff0620-69ef-4db5-90e7-59cd734a204d None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:49:56.108279 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58ff0620-69ef-4db5-90e7-59cd734a204d None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:49:56.108279 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58ff0620-69ef-4db5-90e7-59cd734a204d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:49:56.187502 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-58ff0620-69ef-4db5-90e7-59cd734a204d None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:49:56.188728 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-58ff0620-69ef-4db5-90e7-59cd734a204d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:49:56.253189 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:49:56.304444 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 740/1489] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:49:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 198 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 28 00:49:56.602071 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88eca543-fd98-4341-8643-5951b615ad08 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:49:56.603725 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-88eca543-fd98-4341-8643-5951b615ad08 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:49:56.643596 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:49:56.721051 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88eca543-fd98-4341-8643-5951b615ad08 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:49:56.801692 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9fe9532-aa74-43a5-95d6-13c899c329cc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:49:56.802420 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9fe9532-aa74-43a5-95d6-13c899c329cc service neutron] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:49:56.802748 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9fe9532-aa74-43a5-95d6-13c899c329cc service neutron] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:49:56.803119 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9fe9532-aa74-43a5-95d6-13c899c329cc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:49:56.804994 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-a9fe9532-aa74-43a5-95d6-13c899c329cc service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:49:56.893155 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:49:56.928508 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9fe9532-aa74-43a5-95d6-13c899c329cc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:49:56.930101 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9fe9532-aa74-43a5-95d6-13c899c329cc service neutron] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:49:56.950329 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 750/1490] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:49:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 349 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:49:59.165219 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3256f45-85bd-4f67-b250-07faa66b7c1c None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:49:59.165850 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3256f45-85bd-4f67-b250-07faa66b7c1c None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:49:59.165850 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3256f45-85bd-4f67-b250-07faa66b7c1c None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:49:59.167133 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 741/1491] 10.222.0.42 () {54 vars in 927 bytes} [Sun Aug 28 00:49:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:49:59.173362 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f748ba3-5005-441e-b088-68d93563b942 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:49:59.174243 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f748ba3-5005-441e-b088-68d93563b942 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:49:59.174443 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f748ba3-5005-441e-b088-68d93563b942 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:49:59.251108 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-9f748ba3-5005-441e-b088-68d93563b942 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:49:59.252374 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-9f748ba3-5005-441e-b088-68d93563b942 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:49:59.364489 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 751/1492] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Aug 28 00:49:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:49:59.655976 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3256f45-85bd-4f67-b250-07faa66b7c1c None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:49:59.657775 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-a3256f45-85bd-4f67-b250-07faa66b7c1c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:49:59.806332 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3256f45-85bd-4f67-b250-07faa66b7c1c None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:49:59.889589 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfe85122-624b-4388-aeef-8361ecdd05e0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:49:59.890351 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfe85122-624b-4388-aeef-8361ecdd05e0 service neutron] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:49:59.890674 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfe85122-624b-4388-aeef-8361ecdd05e0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:49:59.890988 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfe85122-624b-4388-aeef-8361ecdd05e0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:49:59.892672 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-bfe85122-624b-4388-aeef-8361ecdd05e0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:49:59.903122 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:50:00.017310 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfe85122-624b-4388-aeef-8361ecdd05e0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:50:00.018976 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfe85122-624b-4388-aeef-8361ecdd05e0 service neutron] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:50:00.039599 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 742/1493] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:49:59 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 384 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 28 00:50:09.362478 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=82 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:50:10.037034 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=50 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:51:13.760156 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f871170-4054-451f-98da-bb47503a0fa0 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:51:13.760845 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f871170-4054-451f-98da-bb47503a0fa0 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:51:13.760845 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f871170-4054-451f-98da-bb47503a0fa0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:51:13.880120 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-9f871170-4054-451f-98da-bb47503a0fa0 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:51:13.882049 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-9f871170-4054-451f-98da-bb47503a0fa0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:51:14.121540 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 752/1494] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:51:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 363 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 28 00:51:14.134806 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d20fdd-9d3d-4c4f-ad72-9857b680d9b8 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:51:14.135408 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d20fdd-9d3d-4c4f-ad72-9857b680d9b8 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:51:14.135408 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d20fdd-9d3d-4c4f-ad72-9857b680d9b8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:51:14.222456 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-d4d20fdd-9d3d-4c4f-ad72-9857b680d9b8 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:51:14.223826 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-d4d20fdd-9d3d-4c4f-ad72-9857b680d9b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:51:14.333846 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 743/1495] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:51:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 200 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:51:14.338992 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f871170-4054-451f-98da-bb47503a0fa0 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:51:14.340415 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-9f871170-4054-451f-98da-bb47503a0fa0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:51:14.352726 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f871170-4054-451f-98da-bb47503a0fa0 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:51:14.418541 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:51:14.455006 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1506e03-5377-4402-99cf-3988e462cb18 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:51:14.455587 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1506e03-5377-4402-99cf-3988e462cb18 admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:51:14.455881 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1506e03-5377-4402-99cf-3988e462cb18 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:51:14.456141 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1506e03-5377-4402-99cf-3988e462cb18 admin admin] PATH_INFO: `/v3/domains` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:51:14.456785 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1506e03-5377-4402-99cf-3988e462cb18 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:51:14.457110 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1506e03-5377-4402-99cf-3988e462cb18 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:51:14.458132 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: WARNING py.warnings [None req-c1506e03-5377-4402-99cf-3988e462cb18 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Aug 28 00:51:14.458132 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: warnings.warn(msg) Aug 28 00:51:14.458132 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: Aug 28 00:51:14.459017 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1506e03-5377-4402-99cf-3988e462cb18 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:51:14.472862 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 753/1496] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Aug 28 00:51:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:51:14.481700 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4d20fdd-9d3d-4c4f-ad72-9857b680d9b8 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:51:14.491797 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4d20fdd-9d3d-4c4f-ad72-9857b680d9b8 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:51:14.558381 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:51:14.604729 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cfe4a87-fa29-41ae-a456-332809c83094 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:51:14.605422 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cfe4a87-fa29-41ae-a456-332809c83094 admin admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:51:14.605694 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cfe4a87-fa29-41ae-a456-332809c83094 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:51:14.605959 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cfe4a87-fa29-41ae-a456-332809c83094 admin admin] PATH_INFO: `/v3/projects` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:51:14.606637 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cfe4a87-fa29-41ae-a456-332809c83094 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:51:14.608479 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: WARNING py.warnings [None req-2cfe4a87-fa29-41ae-a456-332809c83094 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 28 00:51:14.608479 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: warnings.warn(msg) Aug 28 00:51:14.608479 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: Aug 28 00:51:14.609729 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cfe4a87-fa29-41ae-a456-332809c83094 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:51:14.675989 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 744/1497] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 00:51:14 2022] POST /identity/v3/projects => generated 379 bytes in 195 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:51:14.680682 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1506e03-5377-4402-99cf-3988e462cb18 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:51:14.682296 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-c1506e03-5377-4402-99cf-3988e462cb18 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:51:14.723187 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1506e03-5377-4402-99cf-3988e462cb18 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:51:14.835859 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cc12d43-964a-4260-84f8-a1760aac1a33 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:51:14.836602 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc12d43-964a-4260-84f8-a1760aac1a33 admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:51:14.836750 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc12d43-964a-4260-84f8-a1760aac1a33 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:51:14.837016 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc12d43-964a-4260-84f8-a1760aac1a33 admin admin] PATH_INFO: `/v3/users` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:51:14.837692 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cc12d43-964a-4260-84f8-a1760aac1a33 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:51:14.838730 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: WARNING py.warnings [None req-0cc12d43-964a-4260-84f8-a1760aac1a33 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 28 00:51:14.838730 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: warnings.warn(msg) Aug 28 00:51:14.838730 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: Aug 28 00:51:14.839961 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cc12d43-964a-4260-84f8-a1760aac1a33 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:51:14.887376 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 754/1498] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 00:51:14 2022] POST /identity/v3/users => generated 290 bytes in 207 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:51:14.892103 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cfe4a87-fa29-41ae-a456-332809c83094 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:51:14.894266 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-2cfe4a87-fa29-41ae-a456-332809c83094 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:51:14.968150 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cfe4a87-fa29-41ae-a456-332809c83094 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:51:15.069906 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:51:15.085568 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83a72da2-3899-4bb1-8eeb-934be9645b74 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:51:15.086345 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83a72da2-3899-4bb1-8eeb-934be9645b74 admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:51:15.086777 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83a72da2-3899-4bb1-8eeb-934be9645b74 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:51:15.087201 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83a72da2-3899-4bb1-8eeb-934be9645b74 admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:51:15.088221 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83a72da2-3899-4bb1-8eeb-934be9645b74 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:51:15.089479 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: WARNING py.warnings [None req-83a72da2-3899-4bb1-8eeb-934be9645b74 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Aug 28 00:51:15.089479 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: warnings.warn(msg) Aug 28 00:51:15.089479 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: Aug 28 00:51:15.090695 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83a72da2-3899-4bb1-8eeb-934be9645b74 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:51:15.107090 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 745/1499] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:51:14 2022] GET /identity/v3/roles => generated 1420 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 00:51:15.112049 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cc12d43-964a-4260-84f8-a1760aac1a33 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:51:15.113774 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-0cc12d43-964a-4260-84f8-a1760aac1a33 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:51:15.126583 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cc12d43-964a-4260-84f8-a1760aac1a33 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:51:15.157683 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:51:15.249375 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1959c1a5-7545-46f5-935d-b68ca4b8e31a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:51:15.249882 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1959c1a5-7545-46f5-935d-b68ca4b8e31a admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:51:15.250160 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1959c1a5-7545-46f5-935d-b68ca4b8e31a admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:51:15.250432 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1959c1a5-7545-46f5-935d-b68ca4b8e31a admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:51:15.251102 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1959c1a5-7545-46f5-935d-b68ca4b8e31a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:51:15.252132 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: WARNING py.warnings [None req-1959c1a5-7545-46f5-935d-b68ca4b8e31a admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Aug 28 00:51:15.252132 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: warnings.warn(msg) Aug 28 00:51:15.252132 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: Aug 28 00:51:15.253061 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1959c1a5-7545-46f5-935d-b68ca4b8e31a admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:51:15.266772 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 755/1500] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:51:15 2022] GET /identity/v3/roles => generated 1420 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 00:51:15.271666 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83a72da2-3899-4bb1-8eeb-934be9645b74 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:51:15.273499 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-83a72da2-3899-4bb1-8eeb-934be9645b74 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:51:15.309615 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83a72da2-3899-4bb1-8eeb-934be9645b74 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:51:15.412250 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc68bd92-b66b-499b-b682-55d08d00ff78 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:51:15.413121 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc68bd92-b66b-499b-b682-55d08d00ff78 admin admin] REQUEST_METHOD: `PUT` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:51:15.413308 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc68bd92-b66b-499b-b682-55d08d00ff78 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:51:15.413546 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc68bd92-b66b-499b-b682-55d08d00ff78 admin admin] PATH_INFO: `/v3/projects/0b0af8c2c8e34f96ae454087a12d1d1b/users/5c2741911dd9440caeab12ce9cb48ac9/roles/0487ef7e943e4701915cb8ce0f2899d7` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:51:15.441602 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc68bd92-b66b-499b-b682-55d08d00ff78 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0b0af8c2c8e34f96ae454087a12d1d1b, user_id=5c2741911dd9440caeab12ce9cb48ac9, role_id=0487ef7e943e4701915cb8ce0f2899d7)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:51:15.443511 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: WARNING py.warnings [None req-bc68bd92-b66b-499b-b682-55d08d00ff78 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 28 00:51:15.443511 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: warnings.warn(msg) Aug 28 00:51:15.443511 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: Aug 28 00:51:15.447374 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc68bd92-b66b-499b-b682-55d08d00ff78 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:51:15.491209 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 746/1501] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 00:51:15 2022] PUT /identity/v3/projects/0b0af8c2c8e34f96ae454087a12d1d1b/users/5c2741911dd9440caeab12ce9cb48ac9/roles/0487ef7e943e4701915cb8ce0f2899d7 => generated 0 bytes in 220 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 28 00:51:15.505488 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e650acf6-fdea-4718-a3a5-302d736aeea1 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:51:15.505763 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e650acf6-fdea-4718-a3a5-302d736aeea1 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:51:15.505958 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e650acf6-fdea-4718-a3a5-302d736aeea1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:51:15.573181 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-e650acf6-fdea-4718-a3a5-302d736aeea1 None None] MFA Rules not processed for user `5c2741911dd9440caeab12ce9cb48ac9`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:51:15.574407 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-e650acf6-fdea-4718-a3a5-302d736aeea1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:51:15.653761 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 756/1502] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:51:15 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 150 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:51:15.667782 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86defc5f-8e38-477a-8a8b-e042d2a1fad5 None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:51:15.668152 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86defc5f-8e38-477a-8a8b-e042d2a1fad5 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:51:15.668494 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86defc5f-8e38-477a-8a8b-e042d2a1fad5 None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:51:15.670051 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 747/1503] 10.222.0.42 () {54 vars in 964 bytes} [Sun Aug 28 00:51:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:51:15.675550 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f29a8730-78b7-41db-98ea-821d89d592e6 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:51:15.675881 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f29a8730-78b7-41db-98ea-821d89d592e6 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:51:15.676277 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f29a8730-78b7-41db-98ea-821d89d592e6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:51:15.709285 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:51:15.754420 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-f29a8730-78b7-41db-98ea-821d89d592e6 None None] MFA Rules not processed for user `879afc99ebb9402b94fadc6091040c5c`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:51:15.755683 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-f29a8730-78b7-41db-98ea-821d89d592e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:51:15.855500 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 757/1504] 10.222.0.42 () {58 vars in 1061 bytes} [Sun Aug 28 00:51:15 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:51:15.864084 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86defc5f-8e38-477a-8a8b-e042d2a1fad5 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:51:15.866062 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-86defc5f-8e38-477a-8a8b-e042d2a1fad5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:51:15.922916 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:51:16.002433 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86defc5f-8e38-477a-8a8b-e042d2a1fad5 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:51:16.124986 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af172c7a-b1ef-4804-9cd1-d4d28a9d0086 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:51:16.125609 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af172c7a-b1ef-4804-9cd1-d4d28a9d0086 service cinder] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:51:16.125882 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af172c7a-b1ef-4804-9cd1-d4d28a9d0086 service cinder] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:51:16.126146 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af172c7a-b1ef-4804-9cd1-d4d28a9d0086 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:51:16.127764 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-af172c7a-b1ef-4804-9cd1-d4d28a9d0086 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:51:16.273239 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af172c7a-b1ef-4804-9cd1-d4d28a9d0086 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:51:16.275747 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af172c7a-b1ef-4804-9cd1-d4d28a9d0086 service cinder] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:51:16.280552 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:51:16.300083 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 748/1505] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:51:15 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 437 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:51:16.349532 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46152e64-46ac-468e-b8f4-1f1faa741f20 None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:51:16.349883 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46152e64-46ac-468e-b8f4-1f1faa741f20 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:51:16.350199 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46152e64-46ac-468e-b8f4-1f1faa741f20 None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:51:16.351493 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 758/1506] 10.222.0.42 () {52 vars in 830 bytes} [Sun Aug 28 00:51:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 28 00:51:16.355973 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af172c7a-b1ef-4804-9cd1-d4d28a9d0086 service cinder] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:51:16.357293 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-af172c7a-b1ef-4804-9cd1-d4d28a9d0086 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:51:16.467652 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af172c7a-b1ef-4804-9cd1-d4d28a9d0086 service cinder] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:51:16.580582 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30c1a48e-0b7a-46c4-be1f-823fc57072ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:51:16.581241 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30c1a48e-0b7a-46c4-be1f-823fc57072ee admin admin] REQUEST_METHOD: `DELETE` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:51:16.581515 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30c1a48e-0b7a-46c4-be1f-823fc57072ee admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:51:16.581783 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30c1a48e-0b7a-46c4-be1f-823fc57072ee admin admin] PATH_INFO: `/v3/users/5c2741911dd9440caeab12ce9cb48ac9` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:51:16.596179 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30c1a48e-0b7a-46c4-be1f-823fc57072ee admin admin] RBAC: Authorizing `identity:delete_user(user_id=5c2741911dd9440caeab12ce9cb48ac9)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:51:16.597250 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: WARNING py.warnings [None req-30c1a48e-0b7a-46c4-be1f-823fc57072ee admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 28 00:51:16.597250 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: warnings.warn(msg) Aug 28 00:51:16.597250 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: Aug 28 00:51:16.598499 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30c1a48e-0b7a-46c4-be1f-823fc57072ee admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:51:16.609754 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:51:16.676714 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-30c1a48e-0b7a-46c4-be1f-823fc57072ee admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5c2741911dd9440caeab12ce9cb48ac9'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:51:16.706469 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-30c1a48e-0b7a-46c4-be1f-823fc57072ee admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5c2741911dd9440caeab12ce9cb48ac9'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:51:16.717596 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-30c1a48e-0b7a-46c4-be1f-823fc57072ee admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5c2741911dd9440caeab12ce9cb48ac9'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:51:16.736813 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-30c1a48e-0b7a-46c4-be1f-823fc57072ee admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5c2741911dd9440caeab12ce9cb48ac9'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:51:16.737234 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-30c1a48e-0b7a-46c4-be1f-823fc57072ee admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5c2741911dd9440caeab12ce9cb48ac9'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:51:16.743793 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 749/1507] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 00:51:16 2022] DELETE /identity/v3/users/5c2741911dd9440caeab12ce9cb48ac9 => generated 0 bytes in 388 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:51:16.748902 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd7dcd5f-bacd-4ead-9a13-f4e6dc94ff13 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:51:16.749321 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd7dcd5f-bacd-4ead-9a13-f4e6dc94ff13 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:51:16.749732 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd7dcd5f-bacd-4ead-9a13-f4e6dc94ff13 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:51:16.813736 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-fd7dcd5f-bacd-4ead-9a13-f4e6dc94ff13 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:51:16.814986 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-fd7dcd5f-bacd-4ead-9a13-f4e6dc94ff13 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:51:16.903948 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:51:16.919369 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 759/1508] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:51:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:51:16.930283 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30c1a48e-0b7a-46c4-be1f-823fc57072ee admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:51:16.931595 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-30c1a48e-0b7a-46c4-be1f-823fc57072ee admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:51:17.040249 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30c1a48e-0b7a-46c4-be1f-823fc57072ee admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:51:17.100775 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:DELETE=10,keystone:SELECT=89,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:51:17.128386 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdc029c4-38a0-424b-80da-16b449ef8ab0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:51:17.129224 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdc029c4-38a0-424b-80da-16b449ef8ab0 service neutron] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:51:17.129541 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdc029c4-38a0-424b-80da-16b449ef8ab0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:51:17.129849 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdc029c4-38a0-424b-80da-16b449ef8ab0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:51:17.131661 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-bdc029c4-38a0-424b-80da-16b449ef8ab0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:51:17.257975 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdc029c4-38a0-424b-80da-16b449ef8ab0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:51:17.259432 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdc029c4-38a0-424b-80da-16b449ef8ab0 service neutron] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:51:17.280552 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 750/1509] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:51:16 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 351 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 28 00:51:17.687059 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd7dcd5f-bacd-4ead-9a13-f4e6dc94ff13 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:51:17.688718 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-fd7dcd5f-bacd-4ead-9a13-f4e6dc94ff13 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:51:17.707133 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd7dcd5f-bacd-4ead-9a13-f4e6dc94ff13 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:51:17.825777 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2634d522-d57c-4000-839b-8814a2110a53 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:51:17.826873 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2634d522-d57c-4000-839b-8814a2110a53 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:51:17.827204 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2634d522-d57c-4000-839b-8814a2110a53 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:51:17.827539 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2634d522-d57c-4000-839b-8814a2110a53 admin admin] PATH_INFO: `/v3/projects/0b0af8c2c8e34f96ae454087a12d1d1b` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:51:17.839229 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2634d522-d57c-4000-839b-8814a2110a53 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0b0af8c2c8e34f96ae454087a12d1d1b)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:51:17.840349 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: WARNING py.warnings [None req-2634d522-d57c-4000-839b-8814a2110a53 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 28 00:51:17.840349 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: warnings.warn(msg) Aug 28 00:51:17.840349 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: Aug 28 00:51:17.841714 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2634d522-d57c-4000-839b-8814a2110a53 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:51:17.911496 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-2634d522-d57c-4000-839b-8814a2110a53 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0b0af8c2c8e34f96ae454087a12d1d1b', 'request_id': 'req-2634d522-d57c-4000-839b-8814a2110a53'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:51:17.919181 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-2634d522-d57c-4000-839b-8814a2110a53 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0b0af8c2c8e34f96ae454087a12d1d1b', 'request_id': 'req-2634d522-d57c-4000-839b-8814a2110a53'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:51:17.933528 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-2634d522-d57c-4000-839b-8814a2110a53 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0b0af8c2c8e34f96ae454087a12d1d1b', 'request_id': 'req-2634d522-d57c-4000-839b-8814a2110a53'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:51:17.974628 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-2634d522-d57c-4000-839b-8814a2110a53 admin admin] The token cache is being invalidate because project 0b0af8c2c8e34f96ae454087a12d1d1b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81508) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 00:51:17.975020 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-2634d522-d57c-4000-839b-8814a2110a53 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:51:17.975474 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-2634d522-d57c-4000-839b-8814a2110a53 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:51:17.976785 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 760/1510] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 00:51:17 2022] DELETE /identity/v3/projects/0b0af8c2c8e34f96ae454087a12d1d1b => generated 0 bytes in 291 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:51:27.278045 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=62 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:51:27.971910 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=84,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:52:44.930112 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-614a65f2-6120-4709-8520-a62d2b0ff366 None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:52:44.930771 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-614a65f2-6120-4709-8520-a62d2b0ff366 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:52:44.930771 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-614a65f2-6120-4709-8520-a62d2b0ff366 None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:52:44.932027 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 751/1511] 127.0.0.1 () {54 vars in 901 bytes} [Sun Aug 28 00:52:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:52:44.937011 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce915e2-1cbd-43a0-b028-98ffcf91c68e None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:52:44.937494 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce915e2-1cbd-43a0-b028-98ffcf91c68e None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:52:44.937599 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce915e2-1cbd-43a0-b028-98ffcf91c68e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:52:45.034732 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-6ce915e2-1cbd-43a0-b028-98ffcf91c68e None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:52:45.035981 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-6ce915e2-1cbd-43a0-b028-98ffcf91c68e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:52:45.150222 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 761/1512] 127.0.0.1 () {58 vars in 998 bytes} [Sun Aug 28 00:52:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 214 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:52:45.301797 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-614a65f2-6120-4709-8520-a62d2b0ff366 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:52:45.304679 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-614a65f2-6120-4709-8520-a62d2b0ff366 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:52:45.449938 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-614a65f2-6120-4709-8520-a62d2b0ff366 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:52:45.456678 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3d7106e-a5c8-46e2-85e9-88f332175d7e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '168b07df50c24593ba73dbde5b037aed', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:52:45.457351 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3d7106e-a5c8-46e2-85e9-88f332175d7e service nova] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:52:45.457623 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3d7106e-a5c8-46e2-85e9-88f332175d7e service nova] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:52:45.457888 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3d7106e-a5c8-46e2-85e9-88f332175d7e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:52:45.467571 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3d7106e-a5c8-46e2-85e9-88f332175d7e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:52:45.469219 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3d7106e-a5c8-46e2-85e9-88f332175d7e service nova] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:52:45.479335 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 752/1513] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:52:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 178 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:52:46.439879 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e4a405a-6a71-4855-8676-212d60248e33 None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:52:46.440577 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e4a405a-6a71-4855-8676-212d60248e33 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:52:46.440683 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e4a405a-6a71-4855-8676-212d60248e33 None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:52:46.442301 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 762/1514] 127.0.0.1 () {54 vars in 901 bytes} [Sun Aug 28 00:52:46 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:52:46.448326 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf79bf5-02ce-4882-8402-d149c307d507 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:52:46.448951 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf79bf5-02ce-4882-8402-d149c307d507 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:52:46.448951 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf79bf5-02ce-4882-8402-d149c307d507 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:52:46.471640 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-ddf79bf5-02ce-4882-8402-d149c307d507 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:52:46.472860 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-ddf79bf5-02ce-4882-8402-d149c307d507 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:52:46.481093 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 753/1515] 127.0.0.1 () {58 vars in 998 bytes} [Sun Aug 28 00:52:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:52:46.632676 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e4a405a-6a71-4855-8676-212d60248e33 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:52:46.647943 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e4a405a-6a71-4855-8676-212d60248e33 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:52:46.654014 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d532814f-5401-4ca5-8932-130b1810baaa service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '168b07df50c24593ba73dbde5b037aed', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:52:46.654628 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d532814f-5401-4ca5-8932-130b1810baaa service nova] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:52:46.654909 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d532814f-5401-4ca5-8932-130b1810baaa service nova] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:52:46.655180 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d532814f-5401-4ca5-8932-130b1810baaa service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:52:46.664090 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d532814f-5401-4ca5-8932-130b1810baaa service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:52:46.665745 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d532814f-5401-4ca5-8932-130b1810baaa service nova] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:52:46.675912 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 763/1516] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:52:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:52:47.261935 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97a8fd79-736f-46fe-bc4d-a2ca3b0a2493 None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:52:47.262218 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97a8fd79-736f-46fe-bc4d-a2ca3b0a2493 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:52:47.262498 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97a8fd79-736f-46fe-bc4d-a2ca3b0a2493 None None] PATH_INFO: `/v3` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:52:47.263803 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 754/1517] 10.222.0.26 () {54 vars in 925 bytes} [Sun Aug 28 00:52:47 2022] GET /identity/v3 => generated 255 bytes in 3 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:52:47.271384 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b6b4d0e-6071-4e32-84b6-85d2c6b649f8 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:52:47.271671 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b6b4d0e-6071-4e32-84b6-85d2c6b649f8 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:52:47.271926 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b6b4d0e-6071-4e32-84b6-85d2c6b649f8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:52:47.335258 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-5b6b4d0e-6071-4e32-84b6-85d2c6b649f8 None None] MFA Rules not processed for user `c2a3834fa6894dada3291e4781bbd553`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:52:47.336496 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-5b6b4d0e-6071-4e32-84b6-85d2c6b649f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:52:47.354885 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:52:47.413668 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 764/1518] 10.222.0.26 () {58 vars in 1015 bytes} [Sun Aug 28 00:52:47 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:52:56.466603 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=56 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:52:56.527199 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed0a345-61d0-4552-af98-28735c9736fe None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:52:56.527456 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed0a345-61d0-4552-af98-28735c9736fe None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:52:56.527740 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed0a345-61d0-4552-af98-28735c9736fe None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:52:56.529154 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 755/1519] 10.222.0.42 () {54 vars in 961 bytes} [Sun Aug 28 00:52:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:52:56.534709 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50ca46aa-f11f-474b-9cf1-cc053a97dfdc None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:52:56.535225 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50ca46aa-f11f-474b-9cf1-cc053a97dfdc None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:52:56.535327 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50ca46aa-f11f-474b-9cf1-cc053a97dfdc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:52:56.556334 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-50ca46aa-f11f-474b-9cf1-cc053a97dfdc None None] MFA Rules not processed for user `c2a3834fa6894dada3291e4781bbd553`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:52:56.557590 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-50ca46aa-f11f-474b-9cf1-cc053a97dfdc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:52:56.566926 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 765/1520] 10.222.0.42 () {58 vars in 1058 bytes} [Sun Aug 28 00:52:56 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:52:56.575427 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fed0a345-61d0-4552-af98-28735c9736fe None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:52:56.590146 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fed0a345-61d0-4552-af98-28735c9736fe None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:52:56.596243 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6556d31-7b44-4767-8a5a-d1e031ded223 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2a3834fa6894dada3291e4781bbd553', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:52:56.596850 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6556d31-7b44-4767-8a5a-d1e031ded223 service placement] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:52:56.597154 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6556d31-7b44-4767-8a5a-d1e031ded223 service placement] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:52:56.597426 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6556d31-7b44-4767-8a5a-d1e031ded223 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:52:56.606443 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6556d31-7b44-4767-8a5a-d1e031ded223 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:52:56.608023 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6556d31-7b44-4767-8a5a-d1e031ded223 service placement] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:52:56.618152 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 756/1521] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:52:56 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 28 00:52:57.911684 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-751ada18-b226-4c49-a0cb-a23eae011fab None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:52:57.912348 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-751ada18-b226-4c49-a0cb-a23eae011fab None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:52:57.912348 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-751ada18-b226-4c49-a0cb-a23eae011fab None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:52:57.913620 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 766/1522] 127.0.0.1 () {54 vars in 901 bytes} [Sun Aug 28 00:52:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:52:57.919078 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daeeef0f-78c3-44a9-a70e-cfc0eeef50d3 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:52:57.919658 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daeeef0f-78c3-44a9-a70e-cfc0eeef50d3 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:52:57.919658 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daeeef0f-78c3-44a9-a70e-cfc0eeef50d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:52:57.940566 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-daeeef0f-78c3-44a9-a70e-cfc0eeef50d3 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:52:57.941843 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-daeeef0f-78c3-44a9-a70e-cfc0eeef50d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:52:57.950345 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 757/1523] 127.0.0.1 () {58 vars in 998 bytes} [Sun Aug 28 00:52:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:52:58.104544 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-751ada18-b226-4c49-a0cb-a23eae011fab None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:52:58.124000 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-751ada18-b226-4c49-a0cb-a23eae011fab None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:52:58.130051 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-decdaf00-704e-4992-8a9f-6a4b80ba8850 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '168b07df50c24593ba73dbde5b037aed', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:52:58.130662 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-decdaf00-704e-4992-8a9f-6a4b80ba8850 service nova] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:52:58.130937 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-decdaf00-704e-4992-8a9f-6a4b80ba8850 service nova] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:52:58.131183 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-decdaf00-704e-4992-8a9f-6a4b80ba8850 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:52:58.140377 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-decdaf00-704e-4992-8a9f-6a4b80ba8850 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:52:58.142030 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-decdaf00-704e-4992-8a9f-6a4b80ba8850 service nova] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:52:58.151616 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 767/1524] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:52:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 48 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:52:58.965853 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa605d5-e328-4fef-8646-3b6911b4cf1e None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:52:58.966454 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa605d5-e328-4fef-8646-3b6911b4cf1e None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:52:58.966454 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa605d5-e328-4fef-8646-3b6911b4cf1e None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:52:58.967908 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 758/1525] 127.0.0.1 () {54 vars in 901 bytes} [Sun Aug 28 00:52:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:52:58.972950 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-311caf21-ef87-44e8-a481-9b0ee09e9e98 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:52:58.973435 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-311caf21-ef87-44e8-a481-9b0ee09e9e98 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:52:58.973541 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-311caf21-ef87-44e8-a481-9b0ee09e9e98 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:52:58.993904 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-311caf21-ef87-44e8-a481-9b0ee09e9e98 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:52:58.995148 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-311caf21-ef87-44e8-a481-9b0ee09e9e98 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:52:59.004639 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 768/1526] 127.0.0.1 () {58 vars in 998 bytes} [Sun Aug 28 00:52:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:52:59.157306 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faa605d5-e328-4fef-8646-3b6911b4cf1e None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:52:59.173902 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faa605d5-e328-4fef-8646-3b6911b4cf1e None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:52:59.180336 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-327af3dd-7baa-4c78-9c12-3c68c35c5b04 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '168b07df50c24593ba73dbde5b037aed', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:52:59.180943 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-327af3dd-7baa-4c78-9c12-3c68c35c5b04 service nova] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:52:59.181251 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-327af3dd-7baa-4c78-9c12-3c68c35c5b04 service nova] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:52:59.181522 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-327af3dd-7baa-4c78-9c12-3c68c35c5b04 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:52:59.190492 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-327af3dd-7baa-4c78-9c12-3c68c35c5b04 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:52:59.192100 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-327af3dd-7baa-4c78-9c12-3c68c35c5b04 service nova] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:52:59.202685 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 759/1527] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:52:59 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:53:03.034956 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7a90cf5-9df1-4282-b161-37c79d6358b9 None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:53:03.035769 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7a90cf5-9df1-4282-b161-37c79d6358b9 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:53:03.035961 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7a90cf5-9df1-4282-b161-37c79d6358b9 None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:53:03.037455 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 769/1528] 127.0.0.1 () {54 vars in 917 bytes} [Sun Aug 28 00:53:03 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:53:03.043150 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5db9799-35fd-4327-a874-5568658a5bcc None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:53:03.043630 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5db9799-35fd-4327-a874-5568658a5bcc None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:53:03.044048 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5db9799-35fd-4327-a874-5568658a5bcc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:53:03.069093 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-e5db9799-35fd-4327-a874-5568658a5bcc None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:53:03.070323 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-e5db9799-35fd-4327-a874-5568658a5bcc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:53:03.081512 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 760/1529] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Aug 28 00:53:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:53:03.367001 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7a90cf5-9df1-4282-b161-37c79d6358b9 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:53:03.368833 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-a7a90cf5-9df1-4282-b161-37c79d6358b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:53:03.453291 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7a90cf5-9df1-4282-b161-37c79d6358b9 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:53:03.458251 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8e2ee5c-9704-435c-a26b-f5b24d8f684e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:53:03.458871 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8e2ee5c-9704-435c-a26b-f5b24d8f684e service neutron] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:53:03.459147 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8e2ee5c-9704-435c-a26b-f5b24d8f684e service neutron] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:53:03.459415 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8e2ee5c-9704-435c-a26b-f5b24d8f684e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:53:03.470455 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8e2ee5c-9704-435c-a26b-f5b24d8f684e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:53:03.471914 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8e2ee5c-9704-435c-a26b-f5b24d8f684e service neutron] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:53:03.482059 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 770/1530] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:53:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 116 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:53:05.403655 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b783ef8a-ca1f-4241-a6ca-75ba3752813d None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:53:05.404271 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b783ef8a-ca1f-4241-a6ca-75ba3752813d None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:53:05.404271 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b783ef8a-ca1f-4241-a6ca-75ba3752813d None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:53:05.405611 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 761/1531] 127.0.0.1 () {54 vars in 917 bytes} [Sun Aug 28 00:53:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:53:05.411060 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f7367c-aba5-43d5-a15e-7af3539a73b2 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:53:05.411502 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f7367c-aba5-43d5-a15e-7af3539a73b2 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:53:05.411618 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f7367c-aba5-43d5-a15e-7af3539a73b2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:53:05.432615 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-c5f7367c-aba5-43d5-a15e-7af3539a73b2 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:53:05.433892 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-c5f7367c-aba5-43d5-a15e-7af3539a73b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:53:05.442540 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 771/1532] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Aug 28 00:53:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:53:05.726574 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b783ef8a-ca1f-4241-a6ca-75ba3752813d None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:53:05.738974 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b783ef8a-ca1f-4241-a6ca-75ba3752813d None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:53:05.742814 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b20fc66-b15f-40df-a242-badbf6069556 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:53:05.743433 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b20fc66-b15f-40df-a242-badbf6069556 service neutron] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:53:05.743706 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b20fc66-b15f-40df-a242-badbf6069556 service neutron] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:53:05.743988 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b20fc66-b15f-40df-a242-badbf6069556 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:53:05.753553 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b20fc66-b15f-40df-a242-badbf6069556 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:53:05.754997 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b20fc66-b15f-40df-a242-badbf6069556 service neutron] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:53:05.764596 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 762/1533] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:53:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:53:09.055904 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-900f42a1-5537-4b1d-aebc-b168f0afc68d None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:53:09.056501 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-900f42a1-5537-4b1d-aebc-b168f0afc68d None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:53:09.056501 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-900f42a1-5537-4b1d-aebc-b168f0afc68d None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:53:09.057804 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 772/1534] 127.0.0.1 () {54 vars in 917 bytes} [Sun Aug 28 00:53:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:53:09.063972 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8446f9f8-d003-4dae-8fa1-ec100e9c758e None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:53:09.064472 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8446f9f8-d003-4dae-8fa1-ec100e9c758e None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:53:09.064570 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8446f9f8-d003-4dae-8fa1-ec100e9c758e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:53:09.085170 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-8446f9f8-d003-4dae-8fa1-ec100e9c758e None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:53:09.086382 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-8446f9f8-d003-4dae-8fa1-ec100e9c758e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:53:09.094675 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 763/1535] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Aug 28 00:53:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:53:09.645647 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-900f42a1-5537-4b1d-aebc-b168f0afc68d None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:53:09.647639 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-900f42a1-5537-4b1d-aebc-b168f0afc68d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:53:09.716734 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:53:09.732278 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-900f42a1-5537-4b1d-aebc-b168f0afc68d None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:53:09.736294 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7efd15e2-3eed-48cf-abb6-917ac52dd926 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:53:09.736907 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7efd15e2-3eed-48cf-abb6-917ac52dd926 service cinder] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:53:09.737226 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7efd15e2-3eed-48cf-abb6-917ac52dd926 service cinder] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:53:09.737521 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7efd15e2-3eed-48cf-abb6-917ac52dd926 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:53:09.746923 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7efd15e2-3eed-48cf-abb6-917ac52dd926 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:53:09.748431 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7efd15e2-3eed-48cf-abb6-917ac52dd926 service cinder] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:53:09.758307 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 773/1536] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:53:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 113 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:53:11.679332 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dea365b-2677-4333-82ec-952958bb212c None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:53:11.680029 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dea365b-2677-4333-82ec-952958bb212c None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:53:11.680029 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dea365b-2677-4333-82ec-952958bb212c None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:53:11.681945 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 764/1537] 127.0.0.1 () {54 vars in 917 bytes} [Sun Aug 28 00:53:11 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:53:11.687831 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6f89e41-c940-42df-a58d-0ac4d05cc18c None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:53:11.688302 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6f89e41-c940-42df-a58d-0ac4d05cc18c None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:53:11.688424 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6f89e41-c940-42df-a58d-0ac4d05cc18c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:53:11.709200 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-c6f89e41-c940-42df-a58d-0ac4d05cc18c None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:53:11.710415 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-c6f89e41-c940-42df-a58d-0ac4d05cc18c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:53:11.719810 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 774/1538] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Aug 28 00:53:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:53:12.277361 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dea365b-2677-4333-82ec-952958bb212c None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:53:12.279226 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-4dea365b-2677-4333-82ec-952958bb212c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:53:12.293519 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dea365b-2677-4333-82ec-952958bb212c None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:53:12.298499 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f90dc47d-dbd7-45de-ab1f-43a2c3a80ea2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:53:12.299112 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f90dc47d-dbd7-45de-ab1f-43a2c3a80ea2 service cinder] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:53:12.299415 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f90dc47d-dbd7-45de-ab1f-43a2c3a80ea2 service cinder] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:53:12.299705 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f90dc47d-dbd7-45de-ab1f-43a2c3a80ea2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:53:12.310409 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f90dc47d-dbd7-45de-ab1f-43a2c3a80ea2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:53:12.311919 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f90dc47d-dbd7-45de-ab1f-43a2c3a80ea2 service cinder] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:53:12.323033 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 765/1539] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:53:12 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:53:21.704037 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=11 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:53:22.318431 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=33 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:54:37.144545 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee618c0d-7d28-4154-87e4-58f5d8330423 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:37.145701 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee618c0d-7d28-4154-87e4-58f5d8330423 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:37.146201 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee618c0d-7d28-4154-87e4-58f5d8330423 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:37.175063 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-ee618c0d-7d28-4154-87e4-58f5d8330423 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:54:37.176661 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-ee618c0d-7d28-4154-87e4-58f5d8330423 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:54:37.193425 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 775/1540] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:54:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 50 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:54:37.203255 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8927fd16-c215-46de-aa5e-22e80be7dcb0 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:37.203877 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8927fd16-c215-46de-aa5e-22e80be7dcb0 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:37.203877 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8927fd16-c215-46de-aa5e-22e80be7dcb0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:37.233902 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-8927fd16-c215-46de-aa5e-22e80be7dcb0 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:54:37.235143 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-8927fd16-c215-46de-aa5e-22e80be7dcb0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:54:37.249756 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 766/1541] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:54:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 48 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:54:37.255456 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee618c0d-7d28-4154-87e4-58f5d8330423 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:54:37.270413 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee618c0d-7d28-4154-87e4-58f5d8330423 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:54:37.277478 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe622ae5-ac0f-489e-a839-0a028dec07e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:54:37.278102 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe622ae5-ac0f-489e-a839-0a028dec07e7 admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:37.278377 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe622ae5-ac0f-489e-a839-0a028dec07e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:37.278645 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe622ae5-ac0f-489e-a839-0a028dec07e7 admin admin] PATH_INFO: `/v3/domains` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:37.279298 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe622ae5-ac0f-489e-a839-0a028dec07e7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:54:37.279598 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe622ae5-ac0f-489e-a839-0a028dec07e7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:54:37.281264 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe622ae5-ac0f-489e-a839-0a028dec07e7 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:54:37.298407 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 776/1542] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Aug 28 00:54:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:54:37.306727 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8927fd16-c215-46de-aa5e-22e80be7dcb0 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:54:37.316845 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8927fd16-c215-46de-aa5e-22e80be7dcb0 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:54:37.333374 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f5fff14-ac8e-4607-a638-d14bc708af7b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:54:37.334005 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f5fff14-ac8e-4607-a638-d14bc708af7b admin admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:37.334284 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f5fff14-ac8e-4607-a638-d14bc708af7b admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:37.334556 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f5fff14-ac8e-4607-a638-d14bc708af7b admin admin] PATH_INFO: `/v3/projects` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:37.335203 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f5fff14-ac8e-4607-a638-d14bc708af7b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:54:37.337149 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f5fff14-ac8e-4607-a638-d14bc708af7b admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:54:37.399550 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 767/1543] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 00:54:37 2022] POST /identity/v3/projects => generated 391 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:54:37.404809 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe622ae5-ac0f-489e-a839-0a028dec07e7 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:54:37.424875 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe622ae5-ac0f-489e-a839-0a028dec07e7 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:54:37.502382 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72317764-e657-49c4-8fd4-065b6bb26c7d None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:37.504383 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72317764-e657-49c4-8fd4-065b6bb26c7d None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:37.504383 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72317764-e657-49c4-8fd4-065b6bb26c7d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:37.522131 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8a13d65-38b5-4663-9105-581040991335 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:54:37.522756 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8a13d65-38b5-4663-9105-581040991335 admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:37.523199 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8a13d65-38b5-4663-9105-581040991335 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:37.523481 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8a13d65-38b5-4663-9105-581040991335 admin admin] PATH_INFO: `/v3/users` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:37.524124 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8a13d65-38b5-4663-9105-581040991335 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:54:37.526641 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8a13d65-38b5-4663-9105-581040991335 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:54:37.542134 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-72317764-e657-49c4-8fd4-065b6bb26c7d None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:54:37.543356 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-72317764-e657-49c4-8fd4-065b6bb26c7d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:54:37.558752 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 768/1544] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:54:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 57 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:54:37.568278 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11253cd6-67b0-4ade-979a-03cc1825d26f None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:37.568656 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11253cd6-67b0-4ade-979a-03cc1825d26f None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:37.568840 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11253cd6-67b0-4ade-979a-03cc1825d26f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:37.577817 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 777/1545] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 00:54:37 2022] POST /identity/v3/users => generated 296 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:54:37.586012 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8a13d65-38b5-4663-9105-581040991335 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:54:37.603547 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-11253cd6-67b0-4ade-979a-03cc1825d26f None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:54:37.604859 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-11253cd6-67b0-4ade-979a-03cc1825d26f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:54:37.607378 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8a13d65-38b5-4663-9105-581040991335 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:54:37.614957 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec77e9b2-6347-427a-a167-ce9a4ee56de9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:54:37.615655 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec77e9b2-6347-427a-a167-ce9a4ee56de9 admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:37.615940 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec77e9b2-6347-427a-a167-ce9a4ee56de9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:37.616212 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec77e9b2-6347-427a-a167-ce9a4ee56de9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:37.617242 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec77e9b2-6347-427a-a167-ce9a4ee56de9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:54:37.618972 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec77e9b2-6347-427a-a167-ce9a4ee56de9 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:54:37.622154 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 769/1546] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:54:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 55 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:54:37.627287 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11253cd6-67b0-4ade-979a-03cc1825d26f None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:54:37.640526 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 778/1547] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:54:37 2022] GET /identity/v3/roles => generated 1420 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 00:54:37.643414 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11253cd6-67b0-4ade-979a-03cc1825d26f None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:54:37.647175 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec77e9b2-6347-427a-a167-ce9a4ee56de9 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:54:37.655382 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bdcd825-3447-4a36-8142-e33f85f2822f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:54:37.655938 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bdcd825-3447-4a36-8142-e33f85f2822f admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:37.656213 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bdcd825-3447-4a36-8142-e33f85f2822f admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:37.656480 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bdcd825-3447-4a36-8142-e33f85f2822f admin admin] PATH_INFO: `/v3/domains` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:37.657344 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bdcd825-3447-4a36-8142-e33f85f2822f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:54:37.657653 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bdcd825-3447-4a36-8142-e33f85f2822f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:54:37.658824 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: WARNING py.warnings [None req-4bdcd825-3447-4a36-8142-e33f85f2822f admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Aug 28 00:54:37.658824 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: warnings.warn(msg) Aug 28 00:54:37.658824 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: Aug 28 00:54:37.660042 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bdcd825-3447-4a36-8142-e33f85f2822f admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:54:37.661496 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec77e9b2-6347-427a-a167-ce9a4ee56de9 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:54:37.675275 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-805b5b2b-708a-4dfa-adb4-23452fa86f16 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:54:37.675904 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-805b5b2b-708a-4dfa-adb4-23452fa86f16 admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:37.676188 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-805b5b2b-708a-4dfa-adb4-23452fa86f16 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:37.676463 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-805b5b2b-708a-4dfa-adb4-23452fa86f16 admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:37.677136 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-805b5b2b-708a-4dfa-adb4-23452fa86f16 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:54:37.679199 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-805b5b2b-708a-4dfa-adb4-23452fa86f16 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:54:37.685156 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 770/1548] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Aug 28 00:54:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 58 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 28 00:54:37.686552 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a7e983f-c408-48a2-b8f1-a7cb1707c639 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:37.686837 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a7e983f-c408-48a2-b8f1-a7cb1707c639 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:37.687108 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a7e983f-c408-48a2-b8f1-a7cb1707c639 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:37.702956 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 779/1549] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:54:37 2022] GET /identity/v3/roles => generated 1420 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 00:54:37.703834 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-805b5b2b-708a-4dfa-adb4-23452fa86f16 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:54:37.727882 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-805b5b2b-708a-4dfa-adb4-23452fa86f16 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:54:37.739571 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-9a7e983f-c408-48a2-b8f1-a7cb1707c639 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:54:37.739571 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-9a7e983f-c408-48a2-b8f1-a7cb1707c639 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:54:37.739828 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fa4df0c-536b-4aa3-a9e5-7269a0e8f4f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:54:37.740462 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fa4df0c-536b-4aa3-a9e5-7269a0e8f4f8 admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:37.740752 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fa4df0c-536b-4aa3-a9e5-7269a0e8f4f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:37.741046 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fa4df0c-536b-4aa3-a9e5-7269a0e8f4f8 admin admin] PATH_INFO: `/v3/projects` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:37.741801 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fa4df0c-536b-4aa3-a9e5-7269a0e8f4f8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:54:37.742954 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: WARNING py.warnings [None req-7fa4df0c-536b-4aa3-a9e5-7269a0e8f4f8 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 28 00:54:37.742954 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: warnings.warn(msg) Aug 28 00:54:37.742954 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: Aug 28 00:54:37.744172 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fa4df0c-536b-4aa3-a9e5-7269a0e8f4f8 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:54:37.758432 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 771/1550] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:54:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 73 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:54:37.759304 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a7e983f-c408-48a2-b8f1-a7cb1707c639 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:54:37.774382 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a7e983f-c408-48a2-b8f1-a7cb1707c639 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:54:37.793637 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed22edd3-762c-41ce-b928-152e1e5cf6b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:54:37.794320 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed22edd3-762c-41ce-b928-152e1e5cf6b1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:37.794599 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed22edd3-762c-41ce-b928-152e1e5cf6b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:37.794871 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed22edd3-762c-41ce-b928-152e1e5cf6b1 admin admin] PATH_INFO: `/v3/projects/6f3acee4af62483ab1f2cbf3a47b09e1/users/1ab2fcb3759045c7a076e5c8222a421d/roles/0487ef7e943e4701915cb8ce0f2899d7` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:37.799794 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 780/1551] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 00:54:37 2022] POST /identity/v3/projects => generated 385 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:54:37.801330 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a122c6aa-5c7a-461c-8882-aa20e0c07274 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:37.801683 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a122c6aa-5c7a-461c-8882-aa20e0c07274 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:37.802023 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a122c6aa-5c7a-461c-8882-aa20e0c07274 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:37.813407 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed22edd3-762c-41ce-b928-152e1e5cf6b1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6f3acee4af62483ab1f2cbf3a47b09e1, user_id=1ab2fcb3759045c7a076e5c8222a421d, role_id=0487ef7e943e4701915cb8ce0f2899d7)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:54:37.816135 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed22edd3-762c-41ce-b928-152e1e5cf6b1 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:54:37.822777 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-a122c6aa-5c7a-461c-8882-aa20e0c07274 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:54:37.824008 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-a122c6aa-5c7a-461c-8882-aa20e0c07274 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:54:37.837678 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 772/1552] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 00:54:37 2022] PUT /identity/v3/projects/6f3acee4af62483ab1f2cbf3a47b09e1/users/1ab2fcb3759045c7a076e5c8222a421d/roles/0487ef7e943e4701915cb8ce0f2899d7 => generated 0 bytes in 75 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:54:37.837678 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8013bed9-c4ac-4703-bcca-21d68a8f4a54 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:37.837678 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8013bed9-c4ac-4703-bcca-21d68a8f4a54 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:37.837678 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8013bed9-c4ac-4703-bcca-21d68a8f4a54 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:37.861550 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-8013bed9-c4ac-4703-bcca-21d68a8f4a54 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:54:37.862825 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-8013bed9-c4ac-4703-bcca-21d68a8f4a54 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:54:37.887461 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 781/1553] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:54:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:54:37.888855 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f36ab5b9-cfc9-405d-ac73-3aa1b1de3862 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:37.889299 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f36ab5b9-cfc9-405d-ac73-3aa1b1de3862 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:37.889576 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f36ab5b9-cfc9-405d-ac73-3aa1b1de3862 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:37.915068 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-f36ab5b9-cfc9-405d-ac73-3aa1b1de3862 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:54:37.916670 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-f36ab5b9-cfc9-405d-ac73-3aa1b1de3862 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:54:37.927765 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 782/1554] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:54:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:54:37.929023 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b70eb211-5485-4c6e-a13d-2845c5f61812 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:37.929313 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b70eb211-5485-4c6e-a13d-2845c5f61812 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:37.929585 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b70eb211-5485-4c6e-a13d-2845c5f61812 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:37.932153 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 773/1555] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:54:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:54:37.933458 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12c1210f-59bc-455d-8aa2-eeec20238fba None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:37.933743 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12c1210f-59bc-455d-8aa2-eeec20238fba None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:37.934013 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12c1210f-59bc-455d-8aa2-eeec20238fba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:37.953087 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-b70eb211-5485-4c6e-a13d-2845c5f61812 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:54:37.954309 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-b70eb211-5485-4c6e-a13d-2845c5f61812 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:54:37.954472 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-12c1210f-59bc-455d-8aa2-eeec20238fba None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:54:37.955594 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-12c1210f-59bc-455d-8aa2-eeec20238fba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:54:37.962509 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 783/1556] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:54:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:54:37.963269 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b70eb211-5485-4c6e-a13d-2845c5f61812 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:54:37.963607 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 774/1557] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:54:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:54:37.964737 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87c21ee0-07af-4313-b9bd-b42f0cbd920c None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:37.965039 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87c21ee0-07af-4313-b9bd-b42f0cbd920c None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:37.965311 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87c21ee0-07af-4313-b9bd-b42f0cbd920c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:37.979959 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b70eb211-5485-4c6e-a13d-2845c5f61812 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:54:37.983849 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-87c21ee0-07af-4313-b9bd-b42f0cbd920c None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:54:37.984595 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8531a03b-d65e-40dc-9665-af6fe23b553b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:54:37.985094 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-87c21ee0-07af-4313-b9bd-b42f0cbd920c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:54:37.985220 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8531a03b-d65e-40dc-9665-af6fe23b553b admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:37.985501 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8531a03b-d65e-40dc-9665-af6fe23b553b admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:37.985770 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8531a03b-d65e-40dc-9665-af6fe23b553b admin admin] PATH_INFO: `/v3/users` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:37.986453 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8531a03b-d65e-40dc-9665-af6fe23b553b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:54:37.988371 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8531a03b-d65e-40dc-9665-af6fe23b553b admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:54:37.992922 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 775/1558] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:54:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:54:37.993918 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f86ef12-c557-471b-aca7-0d6f804e3982 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:37.994195 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f86ef12-c557-471b-aca7-0d6f804e3982 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:37.994463 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f86ef12-c557-471b-aca7-0d6f804e3982 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:38.006534 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:54:38.017606 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-8f86ef12-c557-471b-aca7-0d6f804e3982 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:54:38.018642 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 784/1559] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 00:54:37 2022] POST /identity/v3/users => generated 293 bytes in 56 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:54:38.018971 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-8f86ef12-c557-471b-aca7-0d6f804e3982 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:54:38.019953 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a888a04-6072-4c33-940d-8d181311ea93 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:38.020588 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a888a04-6072-4c33-940d-8d181311ea93 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:38.021468 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a888a04-6072-4c33-940d-8d181311ea93 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:38.028241 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 776/1560] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:54:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:54:38.029513 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-191a7ef7-fd06-4c22-8016-feccd3251ede None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:38.029790 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-191a7ef7-fd06-4c22-8016-feccd3251ede None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:38.030919 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-191a7ef7-fd06-4c22-8016-feccd3251ede None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:38.041345 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-3a888a04-6072-4c33-940d-8d181311ea93 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:54:38.042561 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-3a888a04-6072-4c33-940d-8d181311ea93 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:54:38.049081 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-191a7ef7-fd06-4c22-8016-feccd3251ede None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:54:38.050247 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-191a7ef7-fd06-4c22-8016-feccd3251ede None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:54:38.050562 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 785/1561] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:54:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:54:38.053169 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a888a04-6072-4c33-940d-8d181311ea93 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:54:38.059159 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 777/1562] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:54:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:54:38.060436 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ace2f89c-c474-408b-9eb3-3590b101b372 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:38.060713 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ace2f89c-c474-408b-9eb3-3590b101b372 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:38.061029 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ace2f89c-c474-408b-9eb3-3590b101b372 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:38.064259 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a888a04-6072-4c33-940d-8d181311ea93 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:54:38.068767 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5517760d-e3c4-4b59-abaf-ed8d4583b7bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:54:38.069395 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5517760d-e3c4-4b59-abaf-ed8d4583b7bc admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:38.069668 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5517760d-e3c4-4b59-abaf-ed8d4583b7bc admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:38.069938 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5517760d-e3c4-4b59-abaf-ed8d4583b7bc admin admin] PATH_INFO: `/v3/domains` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:38.070788 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5517760d-e3c4-4b59-abaf-ed8d4583b7bc admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:54:38.071088 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5517760d-e3c4-4b59-abaf-ed8d4583b7bc admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:54:38.072673 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5517760d-e3c4-4b59-abaf-ed8d4583b7bc admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:54:38.080188 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-ace2f89c-c474-408b-9eb3-3590b101b372 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:54:38.081418 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-ace2f89c-c474-408b-9eb3-3590b101b372 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:54:38.085164 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 786/1563] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Aug 28 00:54:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:54:38.086380 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-791e8929-ef4b-4320-a0f2-adcb8f5482b9 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:38.086661 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-791e8929-ef4b-4320-a0f2-adcb8f5482b9 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:38.086930 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-791e8929-ef4b-4320-a0f2-adcb8f5482b9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:38.093604 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 778/1564] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:54:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:54:38.094335 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ace2f89c-c474-408b-9eb3-3590b101b372 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:54:38.109090 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-791e8929-ef4b-4320-a0f2-adcb8f5482b9 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:54:38.110295 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-791e8929-ef4b-4320-a0f2-adcb8f5482b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:54:38.120695 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 787/1565] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:54:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:54:38.124849 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-791e8929-ef4b-4320-a0f2-adcb8f5482b9 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:54:38.131799 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-791e8929-ef4b-4320-a0f2-adcb8f5482b9 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:54:38.131981 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:54:38.137513 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3251588-1e45-42f5-aff3-e3833dde2849 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:54:38.138136 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3251588-1e45-42f5-aff3-e3833dde2849 admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:38.138544 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3251588-1e45-42f5-aff3-e3833dde2849 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:38.138821 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3251588-1e45-42f5-aff3-e3833dde2849 admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:38.139471 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3251588-1e45-42f5-aff3-e3833dde2849 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:54:38.141154 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3251588-1e45-42f5-aff3-e3833dde2849 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:54:38.151749 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 788/1566] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:54:38 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 00:54:38.155554 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3251588-1e45-42f5-aff3-e3833dde2849 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:54:38.155650 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ace2f89c-c474-408b-9eb3-3590b101b372 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:54:38.157439 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eed9e74-56bf-4c2b-8243-cb3470504b2d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:54:38.158050 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eed9e74-56bf-4c2b-8243-cb3470504b2d service neutron] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:38.158438 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eed9e74-56bf-4c2b-8243-cb3470504b2d service neutron] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:38.158819 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eed9e74-56bf-4c2b-8243-cb3470504b2d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:38.162522 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3251588-1e45-42f5-aff3-e3833dde2849 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:54:38.167472 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fd1f5a0-00f8-447d-8a75-36d9c7c8f716 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:54:38.168229 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eed9e74-56bf-4c2b-8243-cb3470504b2d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:54:38.168382 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fd1f5a0-00f8-447d-8a75-36d9c7c8f716 admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:38.168511 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fd1f5a0-00f8-447d-8a75-36d9c7c8f716 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:38.168787 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fd1f5a0-00f8-447d-8a75-36d9c7c8f716 admin admin] PATH_INFO: `/v3/domains` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:38.169483 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fd1f5a0-00f8-447d-8a75-36d9c7c8f716 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:54:38.169725 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eed9e74-56bf-4c2b-8243-cb3470504b2d service neutron] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:54:38.169841 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fd1f5a0-00f8-447d-8a75-36d9c7c8f716 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:54:38.171349 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fd1f5a0-00f8-447d-8a75-36d9c7c8f716 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:54:38.179265 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 779/1567] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:54:38 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:54:38.180540 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce0562e3-955d-4af8-9113-e29f1dde7eaa None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:38.180824 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce0562e3-955d-4af8-9113-e29f1dde7eaa None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:38.181129 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce0562e3-955d-4af8-9113-e29f1dde7eaa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:38.183919 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 789/1568] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Aug 28 00:54:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:54:38.184522 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fd1f5a0-00f8-447d-8a75-36d9c7c8f716 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:54:38.195529 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fd1f5a0-00f8-447d-8a75-36d9c7c8f716 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:54:38.201566 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-535862f3-f4a2-47f8-bba0-55d5e972eff4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:54:38.202177 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-535862f3-f4a2-47f8-bba0-55d5e972eff4 admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:38.202635 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-535862f3-f4a2-47f8-bba0-55d5e972eff4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:38.203347 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-535862f3-f4a2-47f8-bba0-55d5e972eff4 admin admin] PATH_INFO: `/v3/domains` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:38.203706 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-535862f3-f4a2-47f8-bba0-55d5e972eff4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:54:38.204014 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-535862f3-f4a2-47f8-bba0-55d5e972eff4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:54:38.205716 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-535862f3-f4a2-47f8-bba0-55d5e972eff4 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:54:38.208409 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-ce0562e3-955d-4af8-9113-e29f1dde7eaa None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:54:38.209808 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-ce0562e3-955d-4af8-9113-e29f1dde7eaa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:54:38.226971 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 790/1569] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Aug 28 00:54:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:54:38.226971 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-535862f3-f4a2-47f8-bba0-55d5e972eff4 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:54:38.227297 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 780/1570] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:54:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:54:38.227297 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce0562e3-955d-4af8-9113-e29f1dde7eaa None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:54:38.231950 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-535862f3-f4a2-47f8-bba0-55d5e972eff4 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:54:38.232447 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce0562e3-955d-4af8-9113-e29f1dde7eaa None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:54:38.236842 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0194da55-6c06-4cf4-8cfa-4a870c8051fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:54:38.237499 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0194da55-6c06-4cf4-8cfa-4a870c8051fe admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:38.237646 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa38322b-d0c8-4e47-90dc-363546e6b277 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:54:38.237765 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0194da55-6c06-4cf4-8cfa-4a870c8051fe admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:38.238042 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0194da55-6c06-4cf4-8cfa-4a870c8051fe admin admin] PATH_INFO: `/v3/projects` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:38.238224 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa38322b-d0c8-4e47-90dc-363546e6b277 admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:38.238500 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa38322b-d0c8-4e47-90dc-363546e6b277 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:38.238769 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa38322b-d0c8-4e47-90dc-363546e6b277 admin admin] PATH_INFO: `/v3/domains` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:38.238877 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0194da55-6c06-4cf4-8cfa-4a870c8051fe admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:54:38.239441 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa38322b-d0c8-4e47-90dc-363546e6b277 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:54:38.239739 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa38322b-d0c8-4e47-90dc-363546e6b277 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:54:38.240780 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0194da55-6c06-4cf4-8cfa-4a870c8051fe admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:54:38.241361 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa38322b-d0c8-4e47-90dc-363546e6b277 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:54:38.261833 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 781/1571] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Aug 28 00:54:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:54:38.261833 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa38322b-d0c8-4e47-90dc-363546e6b277 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:54:38.267206 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa38322b-d0c8-4e47-90dc-363546e6b277 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:54:38.275162 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27df4b36-f700-4526-8905-858ed2390ea0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:54:38.275790 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27df4b36-f700-4526-8905-858ed2390ea0 admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:38.276069 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27df4b36-f700-4526-8905-858ed2390ea0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:38.276379 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27df4b36-f700-4526-8905-858ed2390ea0 admin admin] PATH_INFO: `/v3/domains` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:38.277611 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27df4b36-f700-4526-8905-858ed2390ea0 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:54:38.278411 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27df4b36-f700-4526-8905-858ed2390ea0 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:54:38.279967 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 791/1572] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 00:54:38 2022] POST /identity/v3/projects => generated 357 bytes in 59 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:54:38.281035 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0194da55-6c06-4cf4-8cfa-4a870c8051fe admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:54:38.281745 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27df4b36-f700-4526-8905-858ed2390ea0 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:54:38.299958 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 782/1573] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Aug 28 00:54:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:54:38.301173 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27df4b36-f700-4526-8905-858ed2390ea0 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:54:38.313511 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27df4b36-f700-4526-8905-858ed2390ea0 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:54:38.361830 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0194da55-6c06-4cf4-8cfa-4a870c8051fe admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:54:38.367742 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-303405ed-ff97-4ebe-9db8-c8eceb4fe237 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:54:38.368835 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-303405ed-ff97-4ebe-9db8-c8eceb4fe237 admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:38.369400 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-303405ed-ff97-4ebe-9db8-c8eceb4fe237 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:38.369672 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-303405ed-ff97-4ebe-9db8-c8eceb4fe237 admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:38.370324 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-303405ed-ff97-4ebe-9db8-c8eceb4fe237 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:54:38.372145 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-303405ed-ff97-4ebe-9db8-c8eceb4fe237 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:54:38.384712 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-127e6602-59c0-4c7e-91d5-0e7fe99e52f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:54:38.385568 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 792/1574] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:54:38 2022] GET /identity/v3/roles => generated 1420 bytes in 105 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 00:54:38.386454 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-303405ed-ff97-4ebe-9db8-c8eceb4fe237 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:54:38.388797 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-127e6602-59c0-4c7e-91d5-0e7fe99e52f5 admin admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:38.389116 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-127e6602-59c0-4c7e-91d5-0e7fe99e52f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:38.389391 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-127e6602-59c0-4c7e-91d5-0e7fe99e52f5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:38.390052 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-127e6602-59c0-4c7e-91d5-0e7fe99e52f5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:54:38.392000 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-127e6602-59c0-4c7e-91d5-0e7fe99e52f5 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:54:38.398368 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-303405ed-ff97-4ebe-9db8-c8eceb4fe237 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:54:38.409713 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0620a07-d0e5-4824-90c5-3ad0e4c11284 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:54:38.411236 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0620a07-d0e5-4824-90c5-3ad0e4c11284 admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:38.412038 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0620a07-d0e5-4824-90c5-3ad0e4c11284 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:38.412624 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0620a07-d0e5-4824-90c5-3ad0e4c11284 admin admin] PATH_INFO: `/v3/domains` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:38.414137 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0620a07-d0e5-4824-90c5-3ad0e4c11284 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:54:38.414695 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0620a07-d0e5-4824-90c5-3ad0e4c11284 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:54:38.416603 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0620a07-d0e5-4824-90c5-3ad0e4c11284 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:54:38.430347 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 783/1575] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 00:54:38 2022] POST /identity/v3/projects => generated 387 bytes in 129 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:54:38.431195 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 793/1576] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Aug 28 00:54:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:54:38.431682 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-127e6602-59c0-4c7e-91d5-0e7fe99e52f5 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:54:38.432275 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0620a07-d0e5-4824-90c5-3ad0e4c11284 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:54:38.444112 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0620a07-d0e5-4824-90c5-3ad0e4c11284 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:54:38.496557 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:54:38.512349 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdaa549f-017d-4f43-bc27-58c52532cc88 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:54:38.513095 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdaa549f-017d-4f43-bc27-58c52532cc88 admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:38.513382 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdaa549f-017d-4f43-bc27-58c52532cc88 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:38.513651 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdaa549f-017d-4f43-bc27-58c52532cc88 admin admin] PATH_INFO: `/v3/projects` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:38.514293 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdaa549f-017d-4f43-bc27-58c52532cc88 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:54:38.516235 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdaa549f-017d-4f43-bc27-58c52532cc88 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:54:38.517398 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-127e6602-59c0-4c7e-91d5-0e7fe99e52f5 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:54:38.522188 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdde4259-b50b-44af-8796-f5a77bd86ec7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:54:38.522872 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdde4259-b50b-44af-8796-f5a77bd86ec7 admin admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:38.523213 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdde4259-b50b-44af-8796-f5a77bd86ec7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:38.523675 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdde4259-b50b-44af-8796-f5a77bd86ec7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:38.524374 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdde4259-b50b-44af-8796-f5a77bd86ec7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:54:38.526763 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdde4259-b50b-44af-8796-f5a77bd86ec7 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:54:38.558003 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 794/1577] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 00:54:38 2022] POST /identity/v3/projects => generated 375 bytes in 126 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:54:38.562149 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdaa549f-017d-4f43-bc27-58c52532cc88 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:54:38.562705 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 784/1578] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 00:54:38 2022] POST /identity/v3/projects => generated 373 bytes in 131 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:54:38.563843 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdde4259-b50b-44af-8796-f5a77bd86ec7 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:54:38.579010 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:54:38.634738 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdde4259-b50b-44af-8796-f5a77bd86ec7 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:54:38.639789 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0df041df-bee6-43b3-b149-aacd309bb334 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:54:38.640407 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0df041df-bee6-43b3-b149-aacd309bb334 admin admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:38.640730 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0df041df-bee6-43b3-b149-aacd309bb334 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:38.641025 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0df041df-bee6-43b3-b149-aacd309bb334 admin admin] PATH_INFO: `/v3/projects` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:38.642704 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0df041df-bee6-43b3-b149-aacd309bb334 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:54:38.644408 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0df041df-bee6-43b3-b149-aacd309bb334 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:54:38.644742 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdaa549f-017d-4f43-bc27-58c52532cc88 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:54:38.650199 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0af95eec-e969-4f38-9a76-aca652510e4c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:54:38.650745 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0af95eec-e969-4f38-9a76-aca652510e4c admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:38.651176 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0af95eec-e969-4f38-9a76-aca652510e4c admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:38.651633 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0af95eec-e969-4f38-9a76-aca652510e4c admin admin] PATH_INFO: `/v3/users` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:38.652256 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0af95eec-e969-4f38-9a76-aca652510e4c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:54:38.654755 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0af95eec-e969-4f38-9a76-aca652510e4c admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:54:38.681307 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 785/1579] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 00:54:38 2022] POST /identity/v3/projects => generated 377 bytes in 115 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:54:38.681307 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0df041df-bee6-43b3-b149-aacd309bb334 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:54:38.686128 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 795/1580] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 00:54:38 2022] POST /identity/v3/users => generated 279 bytes in 128 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:54:38.687288 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0af95eec-e969-4f38-9a76-aca652510e4c admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:54:38.697910 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0af95eec-e969-4f38-9a76-aca652510e4c admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:54:38.751642 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0df041df-bee6-43b3-b149-aacd309bb334 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:54:38.757322 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cec8d1be-63f0-494b-aa10-84114afa6262 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:54:38.758294 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec8d1be-63f0-494b-aa10-84114afa6262 admin admin] REQUEST_METHOD: `PUT` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:38.758710 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec8d1be-63f0-494b-aa10-84114afa6262 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:38.758990 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec8d1be-63f0-494b-aa10-84114afa6262 admin admin] PATH_INFO: `/v3/projects/c22d5ee0e87f478995d585f2b567542a/users/ab1089ce15ea4dcb8052e0e647af4945/roles/0487ef7e943e4701915cb8ce0f2899d7` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:38.763798 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2fae0dd-3979-4ef8-be9c-b215e7a1b3dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:54:38.764524 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2fae0dd-3979-4ef8-be9c-b215e7a1b3dd admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:38.764913 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2fae0dd-3979-4ef8-be9c-b215e7a1b3dd admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:38.765279 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2fae0dd-3979-4ef8-be9c-b215e7a1b3dd admin admin] PATH_INFO: `/v3/users` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:38.765992 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2fae0dd-3979-4ef8-be9c-b215e7a1b3dd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:54:38.768119 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2fae0dd-3979-4ef8-be9c-b215e7a1b3dd admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:54:38.777676 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cec8d1be-63f0-494b-aa10-84114afa6262 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c22d5ee0e87f478995d585f2b567542a, user_id=ab1089ce15ea4dcb8052e0e647af4945, role_id=0487ef7e943e4701915cb8ce0f2899d7)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:54:38.780653 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cec8d1be-63f0-494b-aa10-84114afa6262 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:54:38.794338 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 796/1581] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 00:54:38 2022] POST /identity/v3/users => generated 294 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:54:38.795306 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2fae0dd-3979-4ef8-be9c-b215e7a1b3dd admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:54:38.798061 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 786/1582] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 00:54:38 2022] PUT /identity/v3/projects/c22d5ee0e87f478995d585f2b567542a/users/ab1089ce15ea4dcb8052e0e647af4945/roles/0487ef7e943e4701915cb8ce0f2899d7 => generated 0 bytes in 119 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:54:38.799441 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cec8d1be-63f0-494b-aa10-84114afa6262 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:54:38.815776 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2fae0dd-3979-4ef8-be9c-b215e7a1b3dd admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:54:38.871803 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cec8d1be-63f0-494b-aa10-84114afa6262 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:54:38.876971 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27c08449-34e2-4424-8464-60ad05476625 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:54:38.877606 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27c08449-34e2-4424-8464-60ad05476625 admin admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:38.877970 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27c08449-34e2-4424-8464-60ad05476625 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:38.878393 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27c08449-34e2-4424-8464-60ad05476625 admin admin] PATH_INFO: `/v3/users` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:38.878647 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:54:38.878988 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27c08449-34e2-4424-8464-60ad05476625 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:54:38.880016 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: WARNING py.warnings [None req-27c08449-34e2-4424-8464-60ad05476625 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 28 00:54:38.880016 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: warnings.warn(msg) Aug 28 00:54:38.880016 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: Aug 28 00:54:38.882410 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27c08449-34e2-4424-8464-60ad05476625 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:54:38.886933 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26b40f2d-f2bc-4d17-9bcb-0e0d8bd65ee2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:54:38.887382 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26b40f2d-f2bc-4d17-9bcb-0e0d8bd65ee2 admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:38.887658 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26b40f2d-f2bc-4d17-9bcb-0e0d8bd65ee2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:38.887925 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26b40f2d-f2bc-4d17-9bcb-0e0d8bd65ee2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:38.888715 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26b40f2d-f2bc-4d17-9bcb-0e0d8bd65ee2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:54:38.891474 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26b40f2d-f2bc-4d17-9bcb-0e0d8bd65ee2 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:54:38.910665 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 787/1583] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 00:54:38 2022] POST /identity/v3/users => generated 288 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:54:38.910665 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27c08449-34e2-4424-8464-60ad05476625 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:54:38.921254 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27c08449-34e2-4424-8464-60ad05476625 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:54:38.926596 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68375b12-ec98-4500-a852-a884763a23bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:54:38.927202 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68375b12-ec98-4500-a852-a884763a23bf admin admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:38.927479 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68375b12-ec98-4500-a852-a884763a23bf admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:38.927757 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68375b12-ec98-4500-a852-a884763a23bf admin admin] PATH_INFO: `/v3/users` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:38.928026 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 797/1584] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 00:54:38 2022] POST /identity/v3/projects => generated 385 bytes in 133 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:54:38.928369 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68375b12-ec98-4500-a852-a884763a23bf admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:54:38.929528 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26b40f2d-f2bc-4d17-9bcb-0e0d8bd65ee2 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:54:38.930312 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68375b12-ec98-4500-a852-a884763a23bf admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:54:38.940671 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:54:38.960059 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 788/1585] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 00:54:38 2022] POST /identity/v3/users => generated 287 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:54:38.962313 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68375b12-ec98-4500-a852-a884763a23bf admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:54:38.973692 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68375b12-ec98-4500-a852-a884763a23bf admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:54:39.018562 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26b40f2d-f2bc-4d17-9bcb-0e0d8bd65ee2 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:54:39.025446 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99b6dda6-9d2f-4944-99bb-c18807383486 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:54:39.026593 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99b6dda6-9d2f-4944-99bb-c18807383486 admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:39.027036 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99b6dda6-9d2f-4944-99bb-c18807383486 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:39.027585 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99b6dda6-9d2f-4944-99bb-c18807383486 admin admin] PATH_INFO: `/v3/users` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:39.028485 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99b6dda6-9d2f-4944-99bb-c18807383486 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:54:39.030714 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99b6dda6-9d2f-4944-99bb-c18807383486 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:54:39.063869 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 798/1586] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 00:54:38 2022] POST /identity/v3/users => generated 289 bytes in 135 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:54:39.065364 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99b6dda6-9d2f-4944-99bb-c18807383486 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:54:39.068256 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c1acf12-76bb-4a8d-9e23-fa3a18d2be98 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:54:39.070334 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c1acf12-76bb-4a8d-9e23-fa3a18d2be98 admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:39.070822 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c1acf12-76bb-4a8d-9e23-fa3a18d2be98 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:39.071369 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c1acf12-76bb-4a8d-9e23-fa3a18d2be98 admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:39.072177 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c1acf12-76bb-4a8d-9e23-fa3a18d2be98 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:54:39.074527 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c1acf12-76bb-4a8d-9e23-fa3a18d2be98 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:54:39.080340 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99b6dda6-9d2f-4944-99bb-c18807383486 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:54:39.088391 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-281f093f-deee-4548-a51e-0f9b83ecb83e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:54:39.089433 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-281f093f-deee-4548-a51e-0f9b83ecb83e admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:39.089873 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-281f093f-deee-4548-a51e-0f9b83ecb83e admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:39.090288 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-281f093f-deee-4548-a51e-0f9b83ecb83e admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:39.091111 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-281f093f-deee-4548-a51e-0f9b83ecb83e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:54:39.092895 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-281f093f-deee-4548-a51e-0f9b83ecb83e admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:54:39.096672 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 789/1587] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:54:38 2022] GET /identity/v3/roles => generated 1420 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 00:54:39.096672 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0e73b6a-5527-492c-8a97-8f4a91e49467 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:39.096672 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0e73b6a-5527-492c-8a97-8f4a91e49467 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:39.096672 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0e73b6a-5527-492c-8a97-8f4a91e49467 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:39.115106 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 799/1588] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:54:39 2022] GET /identity/v3/roles => generated 1420 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 00:54:39.115438 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-281f093f-deee-4548-a51e-0f9b83ecb83e admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:54:39.129610 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-281f093f-deee-4548-a51e-0f9b83ecb83e admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:54:39.135004 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7b95b72-a2ee-4d11-8ddd-a2a2fff1a05a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:54:39.135760 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b95b72-a2ee-4d11-8ddd-a2a2fff1a05a admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:39.136075 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-a0e73b6a-5527-492c-8a97-8f4a91e49467 None None] MFA Rules not processed for user `ab1089ce15ea4dcb8052e0e647af4945`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:54:39.136208 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b95b72-a2ee-4d11-8ddd-a2a2fff1a05a admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:39.136461 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b95b72-a2ee-4d11-8ddd-a2a2fff1a05a admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:39.137298 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7b95b72-a2ee-4d11-8ddd-a2a2fff1a05a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:54:39.137533 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-a0e73b6a-5527-492c-8a97-8f4a91e49467 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:54:39.139631 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7b95b72-a2ee-4d11-8ddd-a2a2fff1a05a admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:54:39.155450 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 800/1589] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:54:39 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 00:54:39.155450 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7b95b72-a2ee-4d11-8ddd-a2a2fff1a05a admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:54:39.167789 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7b95b72-a2ee-4d11-8ddd-a2a2fff1a05a admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:54:39.173691 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63c36a3a-9356-40fd-ac54-df55aa63acbe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:54:39.174479 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63c36a3a-9356-40fd-ac54-df55aa63acbe admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:39.175049 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63c36a3a-9356-40fd-ac54-df55aa63acbe admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:39.175049 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63c36a3a-9356-40fd-ac54-df55aa63acbe admin admin] PATH_INFO: `/v3/users` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:39.175873 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63c36a3a-9356-40fd-ac54-df55aa63acbe admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:54:39.178061 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63c36a3a-9356-40fd-ac54-df55aa63acbe admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:54:39.208278 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 801/1590] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 00:54:39 2022] POST /identity/v3/users => generated 293 bytes in 53 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:54:39.208278 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63c36a3a-9356-40fd-ac54-df55aa63acbe admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:54:39.218245 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 790/1591] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:54:39 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:54:39.223309 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63c36a3a-9356-40fd-ac54-df55aa63acbe admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:54:39.226146 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0e73b6a-5527-492c-8a97-8f4a91e49467 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:54:39.229451 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43f46b88-e380-4122-a02e-df845a5ef009 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:54:39.230530 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43f46b88-e380-4122-a02e-df845a5ef009 admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:39.231079 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43f46b88-e380-4122-a02e-df845a5ef009 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:39.231515 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43f46b88-e380-4122-a02e-df845a5ef009 admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:39.232405 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43f46b88-e380-4122-a02e-df845a5ef009 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:54:39.234369 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43f46b88-e380-4122-a02e-df845a5ef009 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:54:39.240184 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0e73b6a-5527-492c-8a97-8f4a91e49467 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:54:39.246351 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faa4330e-ce16-43c9-b6a9-cee2b3ade9dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:54:39.246997 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa4330e-ce16-43c9-b6a9-cee2b3ade9dd admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:39.247259 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa4330e-ce16-43c9-b6a9-cee2b3ade9dd admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:39.247534 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa4330e-ce16-43c9-b6a9-cee2b3ade9dd admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:39.248182 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faa4330e-ce16-43c9-b6a9-cee2b3ade9dd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:54:39.249857 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faa4330e-ce16-43c9-b6a9-cee2b3ade9dd admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:54:39.252353 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 802/1592] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:54:39 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 00:54:39.253377 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43f46b88-e380-4122-a02e-df845a5ef009 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:54:39.262402 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 791/1593] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:54:39 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 00:54:39.263294 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faa4330e-ce16-43c9-b6a9-cee2b3ade9dd admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:54:39.264453 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43f46b88-e380-4122-a02e-df845a5ef009 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:54:39.270890 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd923358-16cf-4ca6-9729-a3aa927c95e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:54:39.271817 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd923358-16cf-4ca6-9729-a3aa927c95e8 admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:39.272103 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd923358-16cf-4ca6-9729-a3aa927c95e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:39.272647 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd923358-16cf-4ca6-9729-a3aa927c95e8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:39.273318 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd923358-16cf-4ca6-9729-a3aa927c95e8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:54:39.273459 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faa4330e-ce16-43c9-b6a9-cee2b3ade9dd admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:54:39.275038 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd923358-16cf-4ca6-9729-a3aa927c95e8 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:54:39.278602 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-155e6b7d-aa53-475e-95bf-b53578520b80 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:54:39.279564 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-155e6b7d-aa53-475e-95bf-b53578520b80 admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:39.279840 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-155e6b7d-aa53-475e-95bf-b53578520b80 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:39.280116 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-155e6b7d-aa53-475e-95bf-b53578520b80 admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:39.280764 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-155e6b7d-aa53-475e-95bf-b53578520b80 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:54:39.282420 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-155e6b7d-aa53-475e-95bf-b53578520b80 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:54:39.293451 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 803/1594] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:54:39 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 00:54:39.294065 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd923358-16cf-4ca6-9729-a3aa927c95e8 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:54:39.294412 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 792/1595] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:54:39 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 00:54:39.295436 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-155e6b7d-aa53-475e-95bf-b53578520b80 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:54:39.306748 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-155e6b7d-aa53-475e-95bf-b53578520b80 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:54:39.306972 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd923358-16cf-4ca6-9729-a3aa927c95e8 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:54:39.312478 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98e2f30c-9cdd-4c11-8dca-282609aa45b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:54:39.313257 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98e2f30c-9cdd-4c11-8dca-282609aa45b1 admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:39.313665 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98e2f30c-9cdd-4c11-8dca-282609aa45b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:39.314097 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98e2f30c-9cdd-4c11-8dca-282609aa45b1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:39.315024 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98e2f30c-9cdd-4c11-8dca-282609aa45b1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:54:39.315337 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-923c9585-ebe4-4ad3-9c7d-119bd3bd84d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:54:39.316059 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-923c9585-ebe4-4ad3-9c7d-119bd3bd84d7 admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:39.316492 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-923c9585-ebe4-4ad3-9c7d-119bd3bd84d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:39.316815 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-923c9585-ebe4-4ad3-9c7d-119bd3bd84d7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:39.317026 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98e2f30c-9cdd-4c11-8dca-282609aa45b1 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:54:39.317572 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-923c9585-ebe4-4ad3-9c7d-119bd3bd84d7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:54:39.320825 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-923c9585-ebe4-4ad3-9c7d-119bd3bd84d7 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:54:39.333147 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 793/1596] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:54:39 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 00:54:39.334414 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98e2f30c-9cdd-4c11-8dca-282609aa45b1 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:54:39.338652 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 804/1597] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:54:39 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 00:54:39.339523 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-923c9585-ebe4-4ad3-9c7d-119bd3bd84d7 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:54:39.348144 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98e2f30c-9cdd-4c11-8dca-282609aa45b1 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:54:39.353302 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-923c9585-ebe4-4ad3-9c7d-119bd3bd84d7 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:54:39.356427 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9af846e8-f9d9-4480-9a02-70e4761b13d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:54:39.357350 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9af846e8-f9d9-4480-9a02-70e4761b13d9 admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:39.357842 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9af846e8-f9d9-4480-9a02-70e4761b13d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:39.358318 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9af846e8-f9d9-4480-9a02-70e4761b13d9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:39.358684 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30a18528-0545-4578-a696-089b15433735 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:54:39.359287 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30a18528-0545-4578-a696-089b15433735 admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:39.359569 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30a18528-0545-4578-a696-089b15433735 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:39.359810 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9af846e8-f9d9-4480-9a02-70e4761b13d9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:54:39.359955 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30a18528-0545-4578-a696-089b15433735 admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:39.360448 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30a18528-0545-4578-a696-089b15433735 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:54:39.362137 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9af846e8-f9d9-4480-9a02-70e4761b13d9 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:54:39.362340 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30a18528-0545-4578-a696-089b15433735 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:54:39.375570 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 794/1598] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:54:39 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 00:54:39.377300 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38cf4a2f-3847-45c2-ba73-03cb7e3c4ce9 None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:39.377786 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38cf4a2f-3847-45c2-ba73-03cb7e3c4ce9 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:39.378213 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38cf4a2f-3847-45c2-ba73-03cb7e3c4ce9 None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:39.380066 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 795/1599] 10.222.0.42 () {54 vars in 962 bytes} [Sun Aug 28 00:54:39 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:54:39.380401 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 805/1600] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:54:39 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 00:54:39.380802 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38cf4a2f-3847-45c2-ba73-03cb7e3c4ce9 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:54:39.381842 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30a18528-0545-4578-a696-089b15433735 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:54:39.396456 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38cf4a2f-3847-45c2-ba73-03cb7e3c4ce9 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:54:39.396746 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30a18528-0545-4578-a696-089b15433735 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:54:39.402396 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d0e7df3-a62f-41b8-ac13-0e2d4f3d686b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:54:39.402597 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3ba87e9-dba2-4aa7-84f0-303c7563b272 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:54:39.403123 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3ba87e9-dba2-4aa7-84f0-303c7563b272 admin admin] REQUEST_METHOD: `PUT` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:39.403604 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d0e7df3-a62f-41b8-ac13-0e2d4f3d686b admin admin] REQUEST_METHOD: `PUT` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:39.403884 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d0e7df3-a62f-41b8-ac13-0e2d4f3d686b admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:39.404015 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3ba87e9-dba2-4aa7-84f0-303c7563b272 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:39.404157 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d0e7df3-a62f-41b8-ac13-0e2d4f3d686b admin admin] PATH_INFO: `/v3/projects/02fc0fc894b641bfa6b5d5e2c66a7ee4/users/6365894d1845471e953664344e2a1545/roles/0487ef7e943e4701915cb8ce0f2899d7` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:39.404273 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3ba87e9-dba2-4aa7-84f0-303c7563b272 admin admin] PATH_INFO: `/v3/projects/19f069b9ad7a4a5984fa947ad755db0a/users/4894ef8b630e4bc084799c26096f7c66/roles/0487ef7e943e4701915cb8ce0f2899d7` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:39.422025 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3ba87e9-dba2-4aa7-84f0-303c7563b272 admin admin] RBAC: Authorizing `identity:create_grant(project_id=19f069b9ad7a4a5984fa947ad755db0a, user_id=4894ef8b630e4bc084799c26096f7c66, role_id=0487ef7e943e4701915cb8ce0f2899d7)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:54:39.423077 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: WARNING py.warnings [None req-d3ba87e9-dba2-4aa7-84f0-303c7563b272 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 28 00:54:39.423077 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: warnings.warn(msg) Aug 28 00:54:39.423077 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: Aug 28 00:54:39.425112 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3ba87e9-dba2-4aa7-84f0-303c7563b272 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:54:39.428768 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d0e7df3-a62f-41b8-ac13-0e2d4f3d686b admin admin] RBAC: Authorizing `identity:create_grant(project_id=02fc0fc894b641bfa6b5d5e2c66a7ee4, user_id=6365894d1845471e953664344e2a1545, role_id=0487ef7e943e4701915cb8ce0f2899d7)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:54:39.431948 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d0e7df3-a62f-41b8-ac13-0e2d4f3d686b admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:54:39.444099 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 806/1601] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 00:54:39 2022] PUT /identity/v3/projects/19f069b9ad7a4a5984fa947ad755db0a/users/4894ef8b630e4bc084799c26096f7c66/roles/0487ef7e943e4701915cb8ce0f2899d7 => generated 0 bytes in 63 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:54:39.445300 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3ba87e9-dba2-4aa7-84f0-303c7563b272 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:54:39.454755 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 796/1602] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 00:54:39 2022] PUT /identity/v3/projects/02fc0fc894b641bfa6b5d5e2c66a7ee4/users/6365894d1845471e953664344e2a1545/roles/0487ef7e943e4701915cb8ce0f2899d7 => generated 0 bytes in 74 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:54:39.457935 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d0e7df3-a62f-41b8-ac13-0e2d4f3d686b admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:54:39.477449 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:54:39.481053 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:INSERT=2,keystone:SELECT=98 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:54:39.544666 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3ba87e9-dba2-4aa7-84f0-303c7563b272 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:54:39.549197 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d0e7df3-a62f-41b8-ac13-0e2d4f3d686b admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:54:39.551954 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59ea96da-3280-489c-855b-74a72c2f0686 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:54:39.552938 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59ea96da-3280-489c-855b-74a72c2f0686 admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:39.553436 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59ea96da-3280-489c-855b-74a72c2f0686 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:39.553843 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59ea96da-3280-489c-855b-74a72c2f0686 admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:39.555188 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59ea96da-3280-489c-855b-74a72c2f0686 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:54:39.556491 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8638e8b2-a0eb-46c7-a9aa-e20aadb03890 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:54:39.558248 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8638e8b2-a0eb-46c7-a9aa-e20aadb03890 admin admin] REQUEST_METHOD: `PUT` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:39.558397 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59ea96da-3280-489c-855b-74a72c2f0686 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:54:39.558493 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8638e8b2-a0eb-46c7-a9aa-e20aadb03890 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:39.558493 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8638e8b2-a0eb-46c7-a9aa-e20aadb03890 admin admin] PATH_INFO: `/v3/projects/2348c1ddbec1476bacb194214794015d/users/3de5d68f67784e33bdcd19a9cdfed744/roles/0487ef7e943e4701915cb8ce0f2899d7` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:39.580853 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 807/1603] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:54:39 2022] GET /identity/v3/roles => generated 1420 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 00:54:39.582631 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59ea96da-3280-489c-855b-74a72c2f0686 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:54:39.586816 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8638e8b2-a0eb-46c7-a9aa-e20aadb03890 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2348c1ddbec1476bacb194214794015d, user_id=3de5d68f67784e33bdcd19a9cdfed744, role_id=0487ef7e943e4701915cb8ce0f2899d7)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:54:39.592177 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8638e8b2-a0eb-46c7-a9aa-e20aadb03890 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:54:39.597276 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59ea96da-3280-489c-855b-74a72c2f0686 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:54:39.603933 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ea7065e-72c7-4eea-b1fc-d7dcb26a4e6b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:54:39.604615 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ea7065e-72c7-4eea-b1fc-d7dcb26a4e6b admin admin] REQUEST_METHOD: `PUT` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:39.604915 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ea7065e-72c7-4eea-b1fc-d7dcb26a4e6b admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:39.605486 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ea7065e-72c7-4eea-b1fc-d7dcb26a4e6b admin admin] PATH_INFO: `/v3/projects/b56e9e0d8dd94c168bca305994863399/users/5ae53fa7d26844618db6b1ae8c887c89/roles/0487ef7e943e4701915cb8ce0f2899d7` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:39.615721 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 797/1604] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 00:54:39 2022] PUT /identity/v3/projects/2348c1ddbec1476bacb194214794015d/users/3de5d68f67784e33bdcd19a9cdfed744/roles/0487ef7e943e4701915cb8ce0f2899d7 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:54:39.616806 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8638e8b2-a0eb-46c7-a9aa-e20aadb03890 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:54:39.626036 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ea7065e-72c7-4eea-b1fc-d7dcb26a4e6b admin admin] RBAC: Authorizing `identity:create_grant(project_id=b56e9e0d8dd94c168bca305994863399, user_id=5ae53fa7d26844618db6b1ae8c887c89, role_id=0487ef7e943e4701915cb8ce0f2899d7)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:54:39.630830 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ea7065e-72c7-4eea-b1fc-d7dcb26a4e6b admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:54:39.653612 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 808/1605] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 00:54:39 2022] PUT /identity/v3/projects/b56e9e0d8dd94c168bca305994863399/users/5ae53fa7d26844618db6b1ae8c887c89/roles/0487ef7e943e4701915cb8ce0f2899d7 => generated 0 bytes in 72 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:54:39.655892 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11f239c5-88f7-4088-8066-c864826f761c None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:39.656329 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11f239c5-88f7-4088-8066-c864826f761c None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:39.656609 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11f239c5-88f7-4088-8066-c864826f761c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:39.707096 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8638e8b2-a0eb-46c7-a9aa-e20aadb03890 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:54:39.709507 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-11f239c5-88f7-4088-8066-c864826f761c None None] MFA Rules not processed for user `168b07df50c24593ba73dbde5b037aed`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:54:39.710755 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-11f239c5-88f7-4088-8066-c864826f761c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:54:39.773344 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41c55ceb-cfe5-4bd0-85a6-ec12fe31be0d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:54:39.778051 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41c55ceb-cfe5-4bd0-85a6-ec12fe31be0d admin admin] REQUEST_METHOD: `PUT` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:39.778349 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41c55ceb-cfe5-4bd0-85a6-ec12fe31be0d admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:39.778629 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41c55ceb-cfe5-4bd0-85a6-ec12fe31be0d admin admin] PATH_INFO: `/v3/projects/d36dd9ee223f496ebc7b6de74b92227a/users/dd49314743b14266a6a646ffd4d26698/roles/0487ef7e943e4701915cb8ce0f2899d7` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:39.800887 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41c55ceb-cfe5-4bd0-85a6-ec12fe31be0d admin admin] RBAC: Authorizing `identity:create_grant(project_id=d36dd9ee223f496ebc7b6de74b92227a, user_id=dd49314743b14266a6a646ffd4d26698, role_id=0487ef7e943e4701915cb8ce0f2899d7)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:54:39.803843 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41c55ceb-cfe5-4bd0-85a6-ec12fe31be0d admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:54:39.812686 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 809/1606] 10.222.0.42 () {58 vars in 1059 bytes} [Sun Aug 28 00:54:39 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:54:39.814631 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d882701a-8275-4e08-8f99-780f7ccdaedb None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:39.815274 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d882701a-8275-4e08-8f99-780f7ccdaedb None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:39.815777 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d882701a-8275-4e08-8f99-780f7ccdaedb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:39.826688 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 798/1607] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 00:54:39 2022] PUT /identity/v3/projects/d36dd9ee223f496ebc7b6de74b92227a/users/dd49314743b14266a6a646ffd4d26698/roles/0487ef7e943e4701915cb8ce0f2899d7 => generated 0 bytes in 210 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:54:39.828481 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-758bb532-8812-4e99-81a4-3cc220394f4b None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:39.829005 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-758bb532-8812-4e99-81a4-3cc220394f4b None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:39.829385 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-758bb532-8812-4e99-81a4-3cc220394f4b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:39.869292 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-d882701a-8275-4e08-8f99-780f7ccdaedb None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:54:39.870139 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-d882701a-8275-4e08-8f99-780f7ccdaedb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:54:39.872663 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-758bb532-8812-4e99-81a4-3cc220394f4b None None] MFA Rules not processed for user `6365894d1845471e953664344e2a1545`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:54:39.874505 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-758bb532-8812-4e99-81a4-3cc220394f4b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:54:39.889623 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:54:39.956130 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 799/1608] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:54:39 2022] POST /identity/v3/auth/tokens => generated 3234 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:54:39.957938 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 810/1609] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:54:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:54:39.959406 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-887685ae-524f-4519-bf04-3fc822aa047e None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:39.959794 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-887685ae-524f-4519-bf04-3fc822aa047e None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:39.960183 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-887685ae-524f-4519-bf04-3fc822aa047e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:39.964980 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-758bb532-8812-4e99-81a4-3cc220394f4b None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:54:39.978122 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-758bb532-8812-4e99-81a4-3cc220394f4b None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:54:39.983127 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d12a41ad-6ac6-426b-8229-dd0b916e65f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:54:39.983755 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d12a41ad-6ac6-426b-8229-dd0b916e65f3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:39.984030 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d12a41ad-6ac6-426b-8229-dd0b916e65f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:39.984463 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d12a41ad-6ac6-426b-8229-dd0b916e65f3 admin admin] PATH_INFO: `/v3/projects/95df81e14a8549fca0851234f4a00b89/users/d209174d0db04b3584556824c6d0a3d4/roles/0487ef7e943e4701915cb8ce0f2899d7` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:39.985152 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-887685ae-524f-4519-bf04-3fc822aa047e None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:54:39.987064 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-887685ae-524f-4519-bf04-3fc822aa047e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:54:40.000372 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 811/1610] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:54:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:54:40.002115 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36567691-add2-4ad6-afc4-a4e933a1f46a None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:40.002364 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36567691-add2-4ad6-afc4-a4e933a1f46a None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:40.002698 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36567691-add2-4ad6-afc4-a4e933a1f46a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:40.009628 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d12a41ad-6ac6-426b-8229-dd0b916e65f3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=95df81e14a8549fca0851234f4a00b89, user_id=d209174d0db04b3584556824c6d0a3d4, role_id=0487ef7e943e4701915cb8ce0f2899d7)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:54:40.013704 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d12a41ad-6ac6-426b-8229-dd0b916e65f3 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:54:40.029701 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:54:40.033773 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 800/1611] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 00:54:39 2022] PUT /identity/v3/projects/95df81e14a8549fca0851234f4a00b89/users/d209174d0db04b3584556824c6d0a3d4/roles/0487ef7e943e4701915cb8ce0f2899d7 => generated 0 bytes in 77 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:54:40.034865 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d12a41ad-6ac6-426b-8229-dd0b916e65f3 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:54:40.042738 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-36567691-add2-4ad6-afc4-a4e933a1f46a None None] MFA Rules not processed for user `5ae53fa7d26844618db6b1ae8c887c89`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:54:40.044241 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-36567691-add2-4ad6-afc4-a4e933a1f46a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:54:40.169887 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d12a41ad-6ac6-426b-8229-dd0b916e65f3 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:54:40.186557 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c199fcf-5dc6-4458-8edb-ce88d179477b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '168b07df50c24593ba73dbde5b037aed', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:54:40.187296 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c199fcf-5dc6-4458-8edb-ce88d179477b service nova] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:40.187681 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c199fcf-5dc6-4458-8edb-ce88d179477b service nova] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:40.188119 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c199fcf-5dc6-4458-8edb-ce88d179477b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:40.190032 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 812/1612] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:54:40 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:54:40.191336 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e2ebe94-4a29-43bc-b706-23d682c6765e None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:40.191623 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e2ebe94-4a29-43bc-b706-23d682c6765e None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:40.191902 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e2ebe94-4a29-43bc-b706-23d682c6765e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:40.214329 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-3e2ebe94-4a29-43bc-b706-23d682c6765e None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:54:40.215745 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-3e2ebe94-4a29-43bc-b706-23d682c6765e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:54:40.277861 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c199fcf-5dc6-4458-8edb-ce88d179477b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:54:40.281707 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c199fcf-5dc6-4458-8edb-ce88d179477b service nova] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:54:40.284951 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 813/1613] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:54:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:54:40.284951 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e2ebe94-4a29-43bc-b706-23d682c6765e None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:54:40.292816 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e2ebe94-4a29-43bc-b706-23d682c6765e None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:54:40.296400 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 801/1614] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:54:40 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 262 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:54:40.297136 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c199fcf-5dc6-4458-8edb-ce88d179477b service nova] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:54:40.298308 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9898902-ce77-49d6-980d-1039a634dc8d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:54:40.298712 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-6c199fcf-5dc6-4458-8edb-ce88d179477b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:54:40.299370 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9898902-ce77-49d6-980d-1039a634dc8d admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:40.299697 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9898902-ce77-49d6-980d-1039a634dc8d admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:40.300029 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9898902-ce77-49d6-980d-1039a634dc8d admin admin] PATH_INFO: `/v3/projects` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:40.300736 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9898902-ce77-49d6-980d-1039a634dc8d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:54:40.302836 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9898902-ce77-49d6-980d-1039a634dc8d admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:54:40.336830 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 814/1615] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 00:54:40 2022] POST /identity/v3/projects => generated 357 bytes in 53 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:54:40.338176 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9898902-ce77-49d6-980d-1039a634dc8d admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:54:40.342371 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:54:40.383001 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c199fcf-5dc6-4458-8edb-ce88d179477b service nova] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:54:40.409928 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9898902-ce77-49d6-980d-1039a634dc8d admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:54:40.414427 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dffe8e49-cd86-4638-bcb7-7a166c0f04cc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:54:40.415032 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dffe8e49-cd86-4638-bcb7-7a166c0f04cc service neutron] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:40.415306 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dffe8e49-cd86-4638-bcb7-7a166c0f04cc service neutron] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:40.415574 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dffe8e49-cd86-4638-bcb7-7a166c0f04cc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:40.448653 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4198aed0-2c72-47c1-afdb-e21a26b84154 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:54:40.449330 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4198aed0-2c72-47c1-afdb-e21a26b84154 service neutron] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:40.449613 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4198aed0-2c72-47c1-afdb-e21a26b84154 service neutron] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:40.450578 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4198aed0-2c72-47c1-afdb-e21a26b84154 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:40.452360 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:54:40.486483 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dffe8e49-cd86-4638-bcb7-7a166c0f04cc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:54:40.488171 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dffe8e49-cd86-4638-bcb7-7a166c0f04cc service neutron] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:54:40.502507 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 815/1616] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:54:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:54:40.503753 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec00308a-444e-4bbe-a5ab-28c2bd64bbcb None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:40.504033 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec00308a-444e-4bbe-a5ab-28c2bd64bbcb None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:40.504369 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec00308a-444e-4bbe-a5ab-28c2bd64bbcb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:40.524520 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-ec00308a-444e-4bbe-a5ab-28c2bd64bbcb None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:54:40.525779 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-ec00308a-444e-4bbe-a5ab-28c2bd64bbcb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:54:40.537828 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 816/1617] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:54:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:54:40.537828 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec00308a-444e-4bbe-a5ab-28c2bd64bbcb None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:54:40.550257 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec00308a-444e-4bbe-a5ab-28c2bd64bbcb None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:54:40.555603 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fae35728-b1b6-4ad7-b1d6-5fdc124e0025 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:54:40.556215 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae35728-b1b6-4ad7-b1d6-5fdc124e0025 admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:40.556491 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae35728-b1b6-4ad7-b1d6-5fdc124e0025 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:40.556952 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae35728-b1b6-4ad7-b1d6-5fdc124e0025 admin admin] PATH_INFO: `/v3/projects` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:40.557952 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fae35728-b1b6-4ad7-b1d6-5fdc124e0025 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:54:40.560451 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fae35728-b1b6-4ad7-b1d6-5fdc124e0025 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:54:40.563076 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4198aed0-2c72-47c1-afdb-e21a26b84154 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:54:40.565471 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4198aed0-2c72-47c1-afdb-e21a26b84154 service neutron] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:54:40.578249 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 802/1618] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:54:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 281 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:54:40.579131 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4198aed0-2c72-47c1-afdb-e21a26b84154 service neutron] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:54:40.588635 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4198aed0-2c72-47c1-afdb-e21a26b84154 service neutron] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:54:40.592589 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7812caa-4c04-46e6-8509-72ca4e780da9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:54:40.593522 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7812caa-4c04-46e6-8509-72ca4e780da9 service neutron] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:40.593810 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7812caa-4c04-46e6-8509-72ca4e780da9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:40.594085 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7812caa-4c04-46e6-8509-72ca4e780da9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:40.600200 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 817/1619] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 00:54:40 2022] POST /identity/v3/projects => generated 371 bytes in 65 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:54:40.601264 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fae35728-b1b6-4ad7-b1d6-5fdc124e0025 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:54:40.605421 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7812caa-4c04-46e6-8509-72ca4e780da9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:54:40.606778 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7812caa-4c04-46e6-8509-72ca4e780da9 service neutron] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:54:40.619207 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 803/1620] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:54:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:54:40.627179 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7812caa-4c04-46e6-8509-72ca4e780da9 service neutron] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:54:40.645601 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7812caa-4c04-46e6-8509-72ca4e780da9 service neutron] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:54:40.705299 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fae35728-b1b6-4ad7-b1d6-5fdc124e0025 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:54:40.711234 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-424a0a10-9ff8-4d85-8c98-93ff3a2fec4b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:54:40.711905 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-424a0a10-9ff8-4d85-8c98-93ff3a2fec4b admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:40.712186 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-424a0a10-9ff8-4d85-8c98-93ff3a2fec4b admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:40.712453 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-424a0a10-9ff8-4d85-8c98-93ff3a2fec4b admin admin] PATH_INFO: `/v3/users` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:40.713318 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-424a0a10-9ff8-4d85-8c98-93ff3a2fec4b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:54:40.715290 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-424a0a10-9ff8-4d85-8c98-93ff3a2fec4b admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:54:40.729439 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f030bd1c-a582-4197-b532-d909dc00229c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:54:40.730349 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f030bd1c-a582-4197-b532-d909dc00229c service neutron] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:40.730794 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f030bd1c-a582-4197-b532-d909dc00229c service neutron] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:40.731228 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f030bd1c-a582-4197-b532-d909dc00229c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:40.744793 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 818/1621] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 00:54:40 2022] POST /identity/v3/users => generated 285 bytes in 144 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:54:40.745754 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-424a0a10-9ff8-4d85-8c98-93ff3a2fec4b admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:54:40.747972 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f030bd1c-a582-4197-b532-d909dc00229c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:54:40.749652 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f030bd1c-a582-4197-b532-d909dc00229c service neutron] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:54:40.759452 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-424a0a10-9ff8-4d85-8c98-93ff3a2fec4b admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:54:40.766591 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a09ccdb3-3c3d-4ded-8a67-146286518c4a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:54:40.767328 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a09ccdb3-3c3d-4ded-8a67-146286518c4a admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:40.767764 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a09ccdb3-3c3d-4ded-8a67-146286518c4a admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:40.768095 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a09ccdb3-3c3d-4ded-8a67-146286518c4a admin admin] PATH_INFO: `/v3/users` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:40.769303 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a09ccdb3-3c3d-4ded-8a67-146286518c4a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:54:40.772631 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a09ccdb3-3c3d-4ded-8a67-146286518c4a admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:54:40.775605 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 804/1622] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:54:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 156 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:54:40.777145 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95802445-17aa-46c3-9967-96742208f69b None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:40.777593 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95802445-17aa-46c3-9967-96742208f69b None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:40.777878 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95802445-17aa-46c3-9967-96742208f69b None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:40.779328 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 805/1623] 10.222.0.42 () {54 vars in 912 bytes} [Sun Aug 28 00:54:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:54:40.779900 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95802445-17aa-46c3-9967-96742208f69b None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:54:40.800073 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95802445-17aa-46c3-9967-96742208f69b None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:54:40.805458 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5498253-abd9-4467-b3e1-11ad99287126 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:54:40.806102 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5498253-abd9-4467-b3e1-11ad99287126 admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:40.806526 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5498253-abd9-4467-b3e1-11ad99287126 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:40.806819 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5498253-abd9-4467-b3e1-11ad99287126 admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:40.807456 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5498253-abd9-4467-b3e1-11ad99287126 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:54:40.809363 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5498253-abd9-4467-b3e1-11ad99287126 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:54:40.810918 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 819/1624] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 00:54:40 2022] POST /identity/v3/users => generated 292 bytes in 66 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:54:40.818815 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d83c60-3982-4d97-94e9-68807f0a2ee6 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:40.819101 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d83c60-3982-4d97-94e9-68807f0a2ee6 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:40.819380 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d83c60-3982-4d97-94e9-68807f0a2ee6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:40.821166 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 806/1625] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:54:40 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 00:54:40.822047 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5498253-abd9-4467-b3e1-11ad99287126 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:54:40.836635 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5498253-abd9-4467-b3e1-11ad99287126 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:54:40.841940 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88311df2-cfa9-4048-9f51-fb404bb75f8a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:54:40.842570 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88311df2-cfa9-4048-9f51-fb404bb75f8a admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:40.842876 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88311df2-cfa9-4048-9f51-fb404bb75f8a admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:40.843220 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88311df2-cfa9-4048-9f51-fb404bb75f8a admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:40.843875 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88311df2-cfa9-4048-9f51-fb404bb75f8a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:54:40.845940 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88311df2-cfa9-4048-9f51-fb404bb75f8a admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:54:40.853237 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:54:40.861798 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 807/1626] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:54:40 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 00:54:40.862994 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88311df2-cfa9-4048-9f51-fb404bb75f8a admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:54:40.874920 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-c0d83c60-3982-4d97-94e9-68807f0a2ee6 None None] MFA Rules not processed for user `168b07df50c24593ba73dbde5b037aed`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:54:40.876383 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-c0d83c60-3982-4d97-94e9-68807f0a2ee6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:54:40.881177 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88311df2-cfa9-4048-9f51-fb404bb75f8a admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:54:40.888571 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e1faaeb-7948-4fd6-8e2d-9abda9152420 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:54:40.889716 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e1faaeb-7948-4fd6-8e2d-9abda9152420 admin admin] REQUEST_METHOD: `PUT` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:40.889843 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e1faaeb-7948-4fd6-8e2d-9abda9152420 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:40.890179 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e1faaeb-7948-4fd6-8e2d-9abda9152420 admin admin] PATH_INFO: `/v3/projects/5c6f87af024943ee81d82cded504fc67/users/4a475e733cd24983a2a3cecbe7197a0b/roles/f08d6043261747e3b4763c744eef7dfb` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:40.908495 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e1faaeb-7948-4fd6-8e2d-9abda9152420 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5c6f87af024943ee81d82cded504fc67, user_id=4a475e733cd24983a2a3cecbe7197a0b, role_id=f08d6043261747e3b4763c744eef7dfb)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:54:40.912238 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e1faaeb-7948-4fd6-8e2d-9abda9152420 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:54:40.939258 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 808/1627] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 00:54:40 2022] PUT /identity/v3/projects/5c6f87af024943ee81d82cded504fc67/users/4a475e733cd24983a2a3cecbe7197a0b/roles/f08d6043261747e3b4763c744eef7dfb => generated 0 bytes in 77 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:54:40.940129 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e1faaeb-7948-4fd6-8e2d-9abda9152420 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:54:41.055187 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e1faaeb-7948-4fd6-8e2d-9abda9152420 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:54:41.062317 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d01b420-33de-43d0-ab6b-b4bc52a3032a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:54:41.063136 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d01b420-33de-43d0-ab6b-b4bc52a3032a admin admin] REQUEST_METHOD: `PUT` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:41.063482 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d01b420-33de-43d0-ab6b-b4bc52a3032a admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:41.063848 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d01b420-33de-43d0-ab6b-b4bc52a3032a admin admin] PATH_INFO: `/v3/projects/9293a9e25d5a4d43b4c3520f34e96d43/users/97b6ba1a84814d48ae4cff2fd262a854/roles/f08d6043261747e3b4763c744eef7dfb` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:41.081304 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d01b420-33de-43d0-ab6b-b4bc52a3032a admin admin] RBAC: Authorizing `identity:create_grant(project_id=9293a9e25d5a4d43b4c3520f34e96d43, user_id=97b6ba1a84814d48ae4cff2fd262a854, role_id=f08d6043261747e3b4763c744eef7dfb)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:54:41.084393 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d01b420-33de-43d0-ab6b-b4bc52a3032a admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:54:41.117175 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 809/1628] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 00:54:40 2022] PUT /identity/v3/projects/9293a9e25d5a4d43b4c3520f34e96d43/users/97b6ba1a84814d48ae4cff2fd262a854/roles/f08d6043261747e3b4763c744eef7dfb => generated 0 bytes in 177 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:54:41.118410 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d01b420-33de-43d0-ab6b-b4bc52a3032a admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:54:41.121927 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 820/1629] 10.222.0.42 () {58 vars in 1009 bytes} [Sun Aug 28 00:54:40 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 310 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:54:41.123901 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0d83c60-3982-4d97-94e9-68807f0a2ee6 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:54:41.135545 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:54:41.149809 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0d83c60-3982-4d97-94e9-68807f0a2ee6 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:54:41.254114 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d01b420-33de-43d0-ab6b-b4bc52a3032a admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:54:41.268265 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f24e0bd3-3339-4468-8b0f-8478c39a0754 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:54:41.268914 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f24e0bd3-3339-4468-8b0f-8478c39a0754 admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:41.269919 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f24e0bd3-3339-4468-8b0f-8478c39a0754 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:41.270217 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f24e0bd3-3339-4468-8b0f-8478c39a0754 admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:41.273309 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f24e0bd3-3339-4468-8b0f-8478c39a0754 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:54:41.276308 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f24e0bd3-3339-4468-8b0f-8478c39a0754 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:54:41.302147 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a447e70d-105b-45bc-9496-0d9de8ff9956 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:54:41.303039 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a447e70d-105b-45bc-9496-0d9de8ff9956 admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:41.304000 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a447e70d-105b-45bc-9496-0d9de8ff9956 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:41.304716 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a447e70d-105b-45bc-9496-0d9de8ff9956 admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:41.306131 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a447e70d-105b-45bc-9496-0d9de8ff9956 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:54:41.307314 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 810/1630] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:54:41 2022] GET /identity/v3/roles => generated 1420 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 00:54:41.309854 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f24e0bd3-3339-4468-8b0f-8478c39a0754 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:54:41.311283 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a447e70d-105b-45bc-9496-0d9de8ff9956 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:54:41.336546 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 821/1631] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:54:41 2022] GET /identity/v3/roles => generated 1420 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 00:54:41.339137 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a447e70d-105b-45bc-9496-0d9de8ff9956 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:54:41.367005 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a447e70d-105b-45bc-9496-0d9de8ff9956 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:54:41.380451 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81d32ca8-20ed-4c68-a7c6-d811748abcfc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:54:41.381168 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81d32ca8-20ed-4c68-a7c6-d811748abcfc admin admin] REQUEST_METHOD: `PUT` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:41.381449 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81d32ca8-20ed-4c68-a7c6-d811748abcfc admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:41.381724 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81d32ca8-20ed-4c68-a7c6-d811748abcfc admin admin] PATH_INFO: `/v3/projects/5c6f87af024943ee81d82cded504fc67/users/4a475e733cd24983a2a3cecbe7197a0b/roles/f08d6043261747e3b4763c744eef7dfb` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:41.384941 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81d32ca8-20ed-4c68-a7c6-d811748abcfc admin admin] RBAC: Authorizing `identity:create_grant(project_id=5c6f87af024943ee81d82cded504fc67, user_id=4a475e733cd24983a2a3cecbe7197a0b, role_id=f08d6043261747e3b4763c744eef7dfb)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:54:41.387501 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81d32ca8-20ed-4c68-a7c6-d811748abcfc admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:54:41.418964 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 822/1632] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 00:54:41 2022] PUT /identity/v3/projects/5c6f87af024943ee81d82cded504fc67/users/4a475e733cd24983a2a3cecbe7197a0b/roles/f08d6043261747e3b4763c744eef7dfb => generated 0 bytes in 79 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:54:41.426566 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81d32ca8-20ed-4c68-a7c6-d811748abcfc admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:54:41.447795 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f24e0bd3-3339-4468-8b0f-8478c39a0754 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:54:41.468791 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:54:41.593786 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e50d1437-8a35-4648-950b-37cc3b9976ac service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:54:41.594826 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e50d1437-8a35-4648-950b-37cc3b9976ac service cinder] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:41.595110 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e50d1437-8a35-4648-950b-37cc3b9976ac service cinder] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:41.595378 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e50d1437-8a35-4648-950b-37cc3b9976ac service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:41.603388 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81d32ca8-20ed-4c68-a7c6-d811748abcfc admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:54:41.622801 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f3bd2cf-0964-4289-a2f3-afecd4195d77 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:54:41.625290 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f3bd2cf-0964-4289-a2f3-afecd4195d77 admin admin] REQUEST_METHOD: `PUT` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:41.626037 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f3bd2cf-0964-4289-a2f3-afecd4195d77 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:41.628295 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f3bd2cf-0964-4289-a2f3-afecd4195d77 admin admin] PATH_INFO: `/v3/projects/9293a9e25d5a4d43b4c3520f34e96d43/users/97b6ba1a84814d48ae4cff2fd262a854/roles/f08d6043261747e3b4763c744eef7dfb` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:41.633095 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f3bd2cf-0964-4289-a2f3-afecd4195d77 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9293a9e25d5a4d43b4c3520f34e96d43, user_id=97b6ba1a84814d48ae4cff2fd262a854, role_id=f08d6043261747e3b4763c744eef7dfb)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:54:41.638409 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f3bd2cf-0964-4289-a2f3-afecd4195d77 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:54:41.677386 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 823/1633] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 00:54:41 2022] PUT /identity/v3/projects/9293a9e25d5a4d43b4c3520f34e96d43/users/97b6ba1a84814d48ae4cff2fd262a854/roles/f08d6043261747e3b4763c744eef7dfb => generated 0 bytes in 251 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:54:41.677386 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7d86c60-38ff-4239-bb07-7153dc30b18d None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:41.677386 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7d86c60-38ff-4239-bb07-7153dc30b18d None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:41.677386 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7d86c60-38ff-4239-bb07-7153dc30b18d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:41.733649 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-b7d86c60-38ff-4239-bb07-7153dc30b18d None None] MFA Rules not processed for user `4a475e733cd24983a2a3cecbe7197a0b`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:54:41.735276 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-b7d86c60-38ff-4239-bb07-7153dc30b18d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:54:41.762699 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e50d1437-8a35-4648-950b-37cc3b9976ac service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:54:41.764240 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e50d1437-8a35-4648-950b-37cc3b9976ac service cinder] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:54:41.786353 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 811/1634] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:54:41 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 478 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:54:41.787725 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1512306-a273-495c-86d3-a121c9f01323 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:41.788010 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1512306-a273-495c-86d3-a121c9f01323 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:41.788283 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1512306-a273-495c-86d3-a121c9f01323 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:41.850630 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-e1512306-a273-495c-86d3-a121c9f01323 None None] MFA Rules not processed for user `97b6ba1a84814d48ae4cff2fd262a854`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:54:41.851882 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-e1512306-a273-495c-86d3-a121c9f01323 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:54:41.913222 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 824/1635] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:54:41 2022] POST /identity/v3/auth/tokens => generated 3301 bytes in 244 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:54:41.927693 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7d86c60-38ff-4239-bb07-7153dc30b18d None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:54:41.934392 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:54:41.998388 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 812/1636] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:54:41 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 211 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:54:42.071804 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7d86c60-38ff-4239-bb07-7153dc30b18d None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:54:42.077666 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b1497a8-a353-4cb2-a108-9ed1f40e2345 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:54:42.078478 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b1497a8-a353-4cb2-a108-9ed1f40e2345 service cinder] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:42.079393 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b1497a8-a353-4cb2-a108-9ed1f40e2345 service cinder] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:42.079536 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b1497a8-a353-4cb2-a108-9ed1f40e2345 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:42.101322 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b1497a8-a353-4cb2-a108-9ed1f40e2345 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:54:42.102779 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b1497a8-a353-4cb2-a108-9ed1f40e2345 service cinder] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:54:42.121775 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 825/1637] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:54:41 2022] GET /identity/v3/auth/tokens => generated 3301 bytes in 195 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 28 00:54:42.184547 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1512306-a273-495c-86d3-a121c9f01323 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:54:42.195379 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1512306-a273-495c-86d3-a121c9f01323 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:54:42.199327 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebb751ef-827a-46c0-b183-76fb377bde01 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:54:42.200260 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb751ef-827a-46c0-b183-76fb377bde01 service cinder] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:42.200801 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb751ef-827a-46c0-b183-76fb377bde01 service cinder] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:42.201229 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb751ef-827a-46c0-b183-76fb377bde01 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:42.213674 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebb751ef-827a-46c0-b183-76fb377bde01 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:54:42.215259 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebb751ef-827a-46c0-b183-76fb377bde01 service cinder] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:54:42.243199 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 813/1638] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:54:42 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 59 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:54:42.385970 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b1497a8-a353-4cb2-a108-9ed1f40e2345 service cinder] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:54:42.396404 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b1497a8-a353-4cb2-a108-9ed1f40e2345 service cinder] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:54:42.401961 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ce6aeb5-5931-42e6-8be4-6c9905ee352a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:54:42.402864 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce6aeb5-5931-42e6-8be4-6c9905ee352a service cinder] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:42.403265 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce6aeb5-5931-42e6-8be4-6c9905ee352a service cinder] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:42.403638 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce6aeb5-5931-42e6-8be4-6c9905ee352a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:42.481483 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ce6aeb5-5931-42e6-8be4-6c9905ee352a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:54:42.483271 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ce6aeb5-5931-42e6-8be4-6c9905ee352a service cinder] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:54:42.494467 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 826/1639] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:54:42 2022] GET /identity/v3/auth/tokens => generated 3234 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 28 00:54:42.642263 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebb751ef-827a-46c0-b183-76fb377bde01 service cinder] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:54:42.659358 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebb751ef-827a-46c0-b183-76fb377bde01 service cinder] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:54:42.665634 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2365e9fe-ee5a-4d0a-a9da-ddacacc4cef0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:54:42.666270 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2365e9fe-ee5a-4d0a-a9da-ddacacc4cef0 service cinder] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:42.666553 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2365e9fe-ee5a-4d0a-a9da-ddacacc4cef0 service cinder] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:42.666827 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2365e9fe-ee5a-4d0a-a9da-ddacacc4cef0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:42.718687 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:54:42.799517 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2365e9fe-ee5a-4d0a-a9da-ddacacc4cef0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:54:42.801229 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2365e9fe-ee5a-4d0a-a9da-ddacacc4cef0 service cinder] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:54:42.828986 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 814/1640] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:54:42 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 187 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:54:43.674777 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85016b15-8c36-4a7c-8972-a731e8a07cf2 None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:43.675850 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85016b15-8c36-4a7c-8972-a731e8a07cf2 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:43.676305 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85016b15-8c36-4a7c-8972-a731e8a07cf2 None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:43.678202 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 827/1641] 10.222.0.42 () {54 vars in 962 bytes} [Sun Aug 28 00:54:43 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:54:43.686412 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07326304-33a0-4f4c-8380-b1f8509c4b08 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:43.687339 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07326304-33a0-4f4c-8380-b1f8509c4b08 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:43.687823 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07326304-33a0-4f4c-8380-b1f8509c4b08 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:43.723965 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-07326304-33a0-4f4c-8380-b1f8509c4b08 None None] MFA Rules not processed for user `168b07df50c24593ba73dbde5b037aed`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:54:43.725914 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-07326304-33a0-4f4c-8380-b1f8509c4b08 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:54:43.887525 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 815/1642] 10.222.0.42 () {58 vars in 1059 bytes} [Sun Aug 28 00:54:43 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 202 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:54:43.893212 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1721b1a3-94cb-42c5-bb77-d156d6eff94b None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:43.893536 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1721b1a3-94cb-42c5-bb77-d156d6eff94b None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:43.893767 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1721b1a3-94cb-42c5-bb77-d156d6eff94b None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:43.895502 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 828/1643] 10.222.0.42 () {54 vars in 912 bytes} [Sun Aug 28 00:54:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:54:43.901265 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b09bdf96-ee3a-4d9e-bec9-d458b7358fd1 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:43.901636 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b09bdf96-ee3a-4d9e-bec9-d458b7358fd1 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:43.901824 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b09bdf96-ee3a-4d9e-bec9-d458b7358fd1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:43.925278 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-b09bdf96-ee3a-4d9e-bec9-d458b7358fd1 None None] MFA Rules not processed for user `168b07df50c24593ba73dbde5b037aed`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:54:43.926770 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-b09bdf96-ee3a-4d9e-bec9-d458b7358fd1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:54:43.948196 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 816/1644] 10.222.0.42 () {58 vars in 1009 bytes} [Sun Aug 28 00:54:43 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 48 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:54:43.956021 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1721b1a3-94cb-42c5-bb77-d156d6eff94b None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:54:43.984003 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1721b1a3-94cb-42c5-bb77-d156d6eff94b None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:54:43.988712 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e95c5dfc-fd3f-4231-8c7e-a233f599bd91 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:54:43.990141 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e95c5dfc-fd3f-4231-8c7e-a233f599bd91 service cinder] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:43.991120 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e95c5dfc-fd3f-4231-8c7e-a233f599bd91 service cinder] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:43.991609 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e95c5dfc-fd3f-4231-8c7e-a233f599bd91 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:44.005593 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e95c5dfc-fd3f-4231-8c7e-a233f599bd91 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:54:44.007806 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e95c5dfc-fd3f-4231-8c7e-a233f599bd91 service cinder] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:54:44.036023 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 829/1645] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:54:43 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 81 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:54:46.107221 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0b9c751-df3e-4b1b-b14b-21e7efe772ef None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:46.109239 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0b9c751-df3e-4b1b-b14b-21e7efe772ef None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:46.110180 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0b9c751-df3e-4b1b-b14b-21e7efe772ef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:46.157887 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-e0b9c751-df3e-4b1b-b14b-21e7efe772ef None None] MFA Rules not processed for user `1ab2fcb3759045c7a076e5c8222a421d`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:54:46.172930 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-e0b9c751-df3e-4b1b-b14b-21e7efe772ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:54:46.332863 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 817/1646] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:54:46 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 229 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:54:46.347085 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0ac399e-7006-4415-ba8d-d653a00d8ad2 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:46.348266 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0ac399e-7006-4415-ba8d-d653a00d8ad2 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:46.348803 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0ac399e-7006-4415-ba8d-d653a00d8ad2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:46.375037 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-c0ac399e-7006-4415-ba8d-d653a00d8ad2 None None] MFA Rules not processed for user `1ab2fcb3759045c7a076e5c8222a421d`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:54:46.379084 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-c0ac399e-7006-4415-ba8d-d653a00d8ad2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:54:46.402003 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 830/1647] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:54:46 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 56 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:54:46.416950 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0b9c751-df3e-4b1b-b14b-21e7efe772ef None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:54:46.442510 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0b9c751-df3e-4b1b-b14b-21e7efe772ef None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:54:46.454837 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c95ba3d3-b342-4e3e-b406-3f5d3ce036a8 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '168b07df50c24593ba73dbde5b037aed', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:54:46.455479 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c95ba3d3-b342-4e3e-b406-3f5d3ce036a8 service nova] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:46.456461 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c95ba3d3-b342-4e3e-b406-3f5d3ce036a8 service nova] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:46.456461 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c95ba3d3-b342-4e3e-b406-3f5d3ce036a8 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:46.491388 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c95ba3d3-b342-4e3e-b406-3f5d3ce036a8 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:54:46.495503 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c95ba3d3-b342-4e3e-b406-3f5d3ce036a8 service nova] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:54:46.508337 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 818/1648] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:54:46 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:54:47.948432 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c112e2bf-00fe-4698-8cb7-65b40b6644f0 None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:47.949010 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c112e2bf-00fe-4698-8cb7-65b40b6644f0 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:47.949207 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c112e2bf-00fe-4698-8cb7-65b40b6644f0 None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:47.951077 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 831/1649] 10.222.0.42 () {54 vars in 912 bytes} [Sun Aug 28 00:54:47 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:54:47.957467 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c33b857-cdd2-4907-9846-24747c345935 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:47.957966 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c33b857-cdd2-4907-9846-24747c345935 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:47.958049 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c33b857-cdd2-4907-9846-24747c345935 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:48.041611 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-0c33b857-cdd2-4907-9846-24747c345935 None None] MFA Rules not processed for user `364a28c108e744728917ba2b0aa2347d`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:54:48.044678 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-0c33b857-cdd2-4907-9846-24747c345935 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:54:48.062505 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:54:48.157258 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 819/1650] 10.222.0.42 () {58 vars in 1009 bytes} [Sun Aug 28 00:54:47 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 198 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:54:48.179150 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c112e2bf-00fe-4698-8cb7-65b40b6644f0 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:54:48.194342 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c112e2bf-00fe-4698-8cb7-65b40b6644f0 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:54:48.198178 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1699e66-0c8e-4e25-aa24-ce0d23a395ce service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:54:48.198802 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1699e66-0c8e-4e25-aa24-ce0d23a395ce service neutron] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:48.199079 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1699e66-0c8e-4e25-aa24-ce0d23a395ce service neutron] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:48.199349 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1699e66-0c8e-4e25-aa24-ce0d23a395ce service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:48.211103 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1699e66-0c8e-4e25-aa24-ce0d23a395ce service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:54:48.213040 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1699e66-0c8e-4e25-aa24-ce0d23a395ce service neutron] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:54:48.223529 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 832/1651] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:54:48 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:54:49.226005 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c33b857-cdd2-4907-9846-24747c345935 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:54:49.309133 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f189b9bd-d2d7-4bb2-9bc8-6a6b311288cb None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:49.310005 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f189b9bd-d2d7-4bb2-9bc8-6a6b311288cb None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:49.310421 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f189b9bd-d2d7-4bb2-9bc8-6a6b311288cb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:49.352765 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-f189b9bd-d2d7-4bb2-9bc8-6a6b311288cb None None] MFA Rules not processed for user `1ab2fcb3759045c7a076e5c8222a421d`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:54:49.354612 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-f189b9bd-d2d7-4bb2-9bc8-6a6b311288cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:54:49.369995 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 833/1652] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:54:49 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 62 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:54:49.477658 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c33b857-cdd2-4907-9846-24747c345935 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:54:49.496798 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5efb0a6d-8bbf-4209-b7a8-637fbbc21b5d service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2a3834fa6894dada3291e4781bbd553', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:54:49.497861 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5efb0a6d-8bbf-4209-b7a8-637fbbc21b5d service placement] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:49.506048 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5efb0a6d-8bbf-4209-b7a8-637fbbc21b5d service placement] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:49.511937 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5efb0a6d-8bbf-4209-b7a8-637fbbc21b5d service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:49.523525 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-5efb0a6d-8bbf-4209-b7a8-637fbbc21b5d service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:54:49.571980 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5efb0a6d-8bbf-4209-b7a8-637fbbc21b5d service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:54:49.574129 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5efb0a6d-8bbf-4209-b7a8-637fbbc21b5d service placement] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:54:49.599012 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 820/1653] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:54:49 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 374 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:54:53.200804 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63261114-d439-488c-a8db-d60698703c23 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:53.201806 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63261114-d439-488c-a8db-d60698703c23 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:53.202231 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63261114-d439-488c-a8db-d60698703c23 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:53.271946 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-63261114-d439-488c-a8db-d60698703c23 None None] MFA Rules not processed for user `dd49314743b14266a6a646ffd4d26698`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:54:53.273648 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-63261114-d439-488c-a8db-d60698703c23 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:54:53.416223 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:54:53.438837 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 834/1654] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:54:53 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 240 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:54:53.450786 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aca14017-2080-4825-a5b0-66a99970930b None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:53.451318 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aca14017-2080-4825-a5b0-66a99970930b None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:53.451553 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aca14017-2080-4825-a5b0-66a99970930b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:53.493658 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-aca14017-2080-4825-a5b0-66a99970930b None None] MFA Rules not processed for user `dd49314743b14266a6a646ffd4d26698`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:54:53.495242 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-aca14017-2080-4825-a5b0-66a99970930b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:54:53.508561 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 821/1655] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:54:53 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 59 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:54:53.521559 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63261114-d439-488c-a8db-d60698703c23 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:54:53.541771 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63261114-d439-488c-a8db-d60698703c23 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:54:53.567856 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31688f68-3c5d-45af-9f53-720f8f7cbdec service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '168b07df50c24593ba73dbde5b037aed', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:54:53.569350 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31688f68-3c5d-45af-9f53-720f8f7cbdec service nova] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:53.570113 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31688f68-3c5d-45af-9f53-720f8f7cbdec service nova] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:53.570580 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31688f68-3c5d-45af-9f53-720f8f7cbdec service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:53.597873 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31688f68-3c5d-45af-9f53-720f8f7cbdec service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:54:53.599876 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31688f68-3c5d-45af-9f53-720f8f7cbdec service nova] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:54:53.624602 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 835/1656] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:54:53 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:54:53.830572 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cb667ec-86c1-45e5-8144-ad86b2cb02eb None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:53.830966 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cb667ec-86c1-45e5-8144-ad86b2cb02eb None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:53.831354 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cb667ec-86c1-45e5-8144-ad86b2cb02eb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:53.917539 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-6cb667ec-86c1-45e5-8144-ad86b2cb02eb None None] MFA Rules not processed for user `d209174d0db04b3584556824c6d0a3d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:54:53.919384 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-6cb667ec-86c1-45e5-8144-ad86b2cb02eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:54:54.052135 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 822/1657] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:54:53 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 221 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:54:54.270909 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-241b0d24-e698-4bf3-83f6-959eafd071cd None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:54.271434 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-241b0d24-e698-4bf3-83f6-959eafd071cd None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:54.273227 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-241b0d24-e698-4bf3-83f6-959eafd071cd None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:54.274887 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 836/1658] 10.222.0.42 () {54 vars in 912 bytes} [Sun Aug 28 00:54:54 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:54:54.281416 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf5bc636-3bc1-49bc-8a5c-d5d114885309 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:54.282104 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf5bc636-3bc1-49bc-8a5c-d5d114885309 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:54.282617 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf5bc636-3bc1-49bc-8a5c-d5d114885309 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:54.322022 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-bf5bc636-3bc1-49bc-8a5c-d5d114885309 None None] MFA Rules not processed for user `364a28c108e744728917ba2b0aa2347d`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:54:54.323419 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-bf5bc636-3bc1-49bc-8a5c-d5d114885309 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:54:54.345135 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 823/1659] 10.222.0.42 () {58 vars in 1009 bytes} [Sun Aug 28 00:54:54 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 65 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:54:54.363828 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-241b0d24-e698-4bf3-83f6-959eafd071cd None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:54:54.375561 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf5bc636-3bc1-49bc-8a5c-d5d114885309 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:54:54.398901 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf5bc636-3bc1-49bc-8a5c-d5d114885309 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:54:54.401484 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-241b0d24-e698-4bf3-83f6-959eafd071cd None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:54:54.409937 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-739fc4d7-e587-4785-8d33-21d344147cd4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:54:54.411068 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-739fc4d7-e587-4785-8d33-21d344147cd4 service neutron] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:54.411354 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-739fc4d7-e587-4785-8d33-21d344147cd4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:54.412082 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-739fc4d7-e587-4785-8d33-21d344147cd4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:54.415687 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c8447e0-fa7c-4b91-92cc-47bb1f9ff8ae service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '168b07df50c24593ba73dbde5b037aed', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:54:54.416597 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c8447e0-fa7c-4b91-92cc-47bb1f9ff8ae service nova] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:54.420005 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c8447e0-fa7c-4b91-92cc-47bb1f9ff8ae service nova] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:54.428413 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c8447e0-fa7c-4b91-92cc-47bb1f9ff8ae service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:54.431559 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-739fc4d7-e587-4785-8d33-21d344147cd4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:54:54.433063 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-739fc4d7-e587-4785-8d33-21d344147cd4 service neutron] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:54:54.446557 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c8447e0-fa7c-4b91-92cc-47bb1f9ff8ae service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:54:54.451602 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c8447e0-fa7c-4b91-92cc-47bb1f9ff8ae service nova] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:54:54.459406 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 824/1660] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:54:54 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:54:54.467410 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 837/1661] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:54:54 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:54:59.238490 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b794ffc-12f4-41ab-9c50-5909899ccef2 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:59.239334 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b794ffc-12f4-41ab-9c50-5909899ccef2 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:59.239743 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b794ffc-12f4-41ab-9c50-5909899ccef2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:59.253041 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b6bad23-2fdc-481c-95c5-fc18a8a8a405 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:59.253648 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b6bad23-2fdc-481c-95c5-fc18a8a8a405 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:59.253648 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b6bad23-2fdc-481c-95c5-fc18a8a8a405 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:59.328497 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:54:59.335391 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-6b794ffc-12f4-41ab-9c50-5909899ccef2 None None] MFA Rules not processed for user `4894ef8b630e4bc084799c26096f7c66`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:54:59.338636 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-6b794ffc-12f4-41ab-9c50-5909899ccef2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:54:59.412264 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-9b6bad23-2fdc-481c-95c5-fc18a8a8a405 None None] MFA Rules not processed for user `eede08f020e04282bb994b855e371433`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:54:59.415987 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-9b6bad23-2fdc-481c-95c5-fc18a8a8a405 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:54:59.553372 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 825/1662] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:54:59 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 306 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:54:59.559305 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b794ffc-12f4-41ab-9c50-5909899ccef2 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:54:59.581599 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 838/1663] 10.222.0.42 () {58 vars in 1009 bytes} [Sun Aug 28 00:54:59 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 329 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 28 00:54:59.594440 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b794ffc-12f4-41ab-9c50-5909899ccef2 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:54:59.611245 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-409c8083-58b3-4173-8423-89c30690108a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '168b07df50c24593ba73dbde5b037aed', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:54:59.612104 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-409c8083-58b3-4173-8423-89c30690108a service nova] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:54:59.612573 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-409c8083-58b3-4173-8423-89c30690108a service nova] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:54:59.613510 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-409c8083-58b3-4173-8423-89c30690108a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:54:59.632925 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-409c8083-58b3-4173-8423-89c30690108a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:54:59.636825 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-409c8083-58b3-4173-8423-89c30690108a service nova] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:54:59.663879 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 826/1664] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:54:59 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:54:59.772349 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b6bad23-2fdc-481c-95c5-fc18a8a8a405 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:54:59.775374 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-9b6bad23-2fdc-481c-95c5-fc18a8a8a405 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:54:59.994193 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b6bad23-2fdc-481c-95c5-fc18a8a8a405 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:54:59.999437 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d706c2f-5d6b-41ac-b75e-f3ddcc82dc21 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac4c5584582f4f55a7085af4cbbabf9b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:00.000705 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d706c2f-5d6b-41ac-b75e-f3ddcc82dc21 service swift] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:00.001055 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d706c2f-5d6b-41ac-b75e-f3ddcc82dc21 service swift] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:00.001367 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d706c2f-5d6b-41ac-b75e-f3ddcc82dc21 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:00.019227 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d706c2f-5d6b-41ac-b75e-f3ddcc82dc21 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:00.023413 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d706c2f-5d6b-41ac-b75e-f3ddcc82dc21 service swift] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:00.051809 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 839/1665] 10.222.0.42 () {60 vars in 1384 bytes} [Sun Aug 28 00:54:59 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 280 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 28 00:55:00.123322 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1da0dfd9-eaac-488a-8bdd-dbba6a80a0e3 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:00.123607 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1da0dfd9-eaac-488a-8bdd-dbba6a80a0e3 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:00.123879 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1da0dfd9-eaac-488a-8bdd-dbba6a80a0e3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:00.181845 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d706c2f-5d6b-41ac-b75e-f3ddcc82dc21 service swift] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:00.201471 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-1da0dfd9-eaac-488a-8bdd-dbba6a80a0e3 None None] MFA Rules not processed for user `d209174d0db04b3584556824c6d0a3d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:55:00.202643 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-1da0dfd9-eaac-488a-8bdd-dbba6a80a0e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:00.217781 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:55:00.238150 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 827/1666] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:55:00 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:55:00.303141 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d706c2f-5d6b-41ac-b75e-f3ddcc82dc21 service swift] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:00.309610 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b3a164b-3ae7-4dbb-b32b-63a35c335f91 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:00.310478 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b3a164b-3ae7-4dbb-b32b-63a35c335f91 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:00.310908 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b3a164b-3ae7-4dbb-b32b-63a35c335f91 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:00.311310 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b3a164b-3ae7-4dbb-b32b-63a35c335f91 admin admin] PATH_INFO: `/v3/users/ab1089ce15ea4dcb8052e0e647af4945` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:00.312870 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b3a164b-3ae7-4dbb-b32b-63a35c335f91 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ab1089ce15ea4dcb8052e0e647af4945)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:00.314449 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: WARNING py.warnings [None req-6b3a164b-3ae7-4dbb-b32b-63a35c335f91 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 28 00:55:00.314449 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: warnings.warn(msg) Aug 28 00:55:00.314449 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: Aug 28 00:55:00.316240 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b3a164b-3ae7-4dbb-b32b-63a35c335f91 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:00.438710 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-6b3a164b-3ae7-4dbb-b32b-63a35c335f91 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ab1089ce15ea4dcb8052e0e647af4945'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:00.442259 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-6b3a164b-3ae7-4dbb-b32b-63a35c335f91 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ab1089ce15ea4dcb8052e0e647af4945'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:00.465625 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-6b3a164b-3ae7-4dbb-b32b-63a35c335f91 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ab1089ce15ea4dcb8052e0e647af4945'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:00.507140 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-6b3a164b-3ae7-4dbb-b32b-63a35c335f91 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ab1089ce15ea4dcb8052e0e647af4945'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:00.509613 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-6b3a164b-3ae7-4dbb-b32b-63a35c335f91 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ab1089ce15ea4dcb8052e0e647af4945'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:00.674207 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 840/1667] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 00:55:00 2022] DELETE /identity/v3/users/ab1089ce15ea4dcb8052e0e647af4945 => generated 0 bytes in 506 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:55:00.676138 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e2837e3-8c03-49e6-983d-08c9112520ac None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:00.676727 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e2837e3-8c03-49e6-983d-08c9112520ac None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:00.676830 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e2837e3-8c03-49e6-983d-08c9112520ac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:00.753517 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-8e2837e3-8c03-49e6-983d-08c9112520ac None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:55:00.759775 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-8e2837e3-8c03-49e6-983d-08c9112520ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:01.146760 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 828/1668] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:55:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 472 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:55:01.203440 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b3a164b-3ae7-4dbb-b32b-63a35c335f91 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:01.205243 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-6b3a164b-3ae7-4dbb-b32b-63a35c335f91 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:01.520058 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b3a164b-3ae7-4dbb-b32b-63a35c335f91 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:01.530543 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad9cd67a-39f9-4354-a66e-22dc4c3003c7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:01.531517 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad9cd67a-39f9-4354-a66e-22dc4c3003c7 service neutron] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:01.531942 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad9cd67a-39f9-4354-a66e-22dc4c3003c7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:01.532388 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad9cd67a-39f9-4354-a66e-22dc4c3003c7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:01.583432 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad9cd67a-39f9-4354-a66e-22dc4c3003c7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:01.585862 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad9cd67a-39f9-4354-a66e-22dc4c3003c7 service neutron] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:01.604413 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 841/1669] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:55:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 401 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:55:02.411686 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afa7cdbf-17e4-4485-b4a4-78c810cbe92f None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:02.412193 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afa7cdbf-17e4-4485-b4a4-78c810cbe92f None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:02.412272 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afa7cdbf-17e4-4485-b4a4-78c810cbe92f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:02.492756 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-afa7cdbf-17e4-4485-b4a4-78c810cbe92f None None] MFA Rules not processed for user `3de5d68f67784e33bdcd19a9cdfed744`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:55:02.498775 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-afa7cdbf-17e4-4485-b4a4-78c810cbe92f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:02.761603 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 829/1670] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:55:02 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 351 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:55:02.828472 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad9cd67a-39f9-4354-a66e-22dc4c3003c7 service neutron] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:02.832797 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-ad9cd67a-39f9-4354-a66e-22dc4c3003c7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:02.936803 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:55:03.037802 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad9cd67a-39f9-4354-a66e-22dc4c3003c7 service neutron] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:03.065873 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c60057f5-0b4f-44f9-9e0d-ba4fd1cf6e26 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '168b07df50c24593ba73dbde5b037aed', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:03.068358 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c60057f5-0b4f-44f9-9e0d-ba4fd1cf6e26 service nova] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:03.068819 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c60057f5-0b4f-44f9-9e0d-ba4fd1cf6e26 service nova] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:03.069327 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c60057f5-0b4f-44f9-9e0d-ba4fd1cf6e26 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:03.103677 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c60057f5-0b4f-44f9-9e0d-ba4fd1cf6e26 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:03.112802 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c60057f5-0b4f-44f9-9e0d-ba4fd1cf6e26 service nova] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:03.149754 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 842/1671] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:55:02 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 322 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:55:03.505665 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afa7cdbf-17e4-4485-b4a4-78c810cbe92f None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:03.509022 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-afa7cdbf-17e4-4485-b4a4-78c810cbe92f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:03.572919 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afa7cdbf-17e4-4485-b4a4-78c810cbe92f None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:03.631953 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cd0b362-77e9-41fb-887a-13a09c9fd588 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:03.650115 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cd0b362-77e9-41fb-887a-13a09c9fd588 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:03.655337 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cd0b362-77e9-41fb-887a-13a09c9fd588 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:03.656643 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cd0b362-77e9-41fb-887a-13a09c9fd588 admin admin] PATH_INFO: `/v3/projects/c22d5ee0e87f478995d585f2b567542a` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:03.660470 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cd0b362-77e9-41fb-887a-13a09c9fd588 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c22d5ee0e87f478995d585f2b567542a)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:03.665787 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: WARNING py.warnings [None req-2cd0b362-77e9-41fb-887a-13a09c9fd588 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Aug 28 00:55:03.665787 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: warnings.warn(msg) Aug 28 00:55:03.665787 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: Aug 28 00:55:03.675410 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cd0b362-77e9-41fb-887a-13a09c9fd588 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:03.735170 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:55:03.892606 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-2cd0b362-77e9-41fb-887a-13a09c9fd588 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c22d5ee0e87f478995d585f2b567542a', 'request_id': 'req-2cd0b362-77e9-41fb-887a-13a09c9fd588'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:03.913464 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-2cd0b362-77e9-41fb-887a-13a09c9fd588 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c22d5ee0e87f478995d585f2b567542a', 'request_id': 'req-2cd0b362-77e9-41fb-887a-13a09c9fd588'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:03.966838 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-2cd0b362-77e9-41fb-887a-13a09c9fd588 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c22d5ee0e87f478995d585f2b567542a', 'request_id': 'req-2cd0b362-77e9-41fb-887a-13a09c9fd588'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:04.000187 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-2cd0b362-77e9-41fb-887a-13a09c9fd588 admin admin] The token cache is being invalidate because project c22d5ee0e87f478995d585f2b567542a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81507) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 00:55:04.000909 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-2cd0b362-77e9-41fb-887a-13a09c9fd588 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:04.005563 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-2cd0b362-77e9-41fb-887a-13a09c9fd588 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:04.013770 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 830/1672] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 00:55:03 2022] DELETE /identity/v3/projects/c22d5ee0e87f478995d585f2b567542a => generated 0 bytes in 507 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:55:04.021006 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97308cb5-70f3-42ae-ad02-0342f8184668 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:04.021582 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97308cb5-70f3-42ae-ad02-0342f8184668 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:04.021582 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97308cb5-70f3-42ae-ad02-0342f8184668 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:04.090959 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-97308cb5-70f3-42ae-ad02-0342f8184668 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:55:04.113128 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-97308cb5-70f3-42ae-ad02-0342f8184668 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:04.331171 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 843/1673] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:55:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 317 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:55:04.338986 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94de5065-ca18-4779-a74f-5302a66c8ca1 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:04.340533 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94de5065-ca18-4779-a74f-5302a66c8ca1 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:04.341558 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94de5065-ca18-4779-a74f-5302a66c8ca1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:04.421488 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-94de5065-ca18-4779-a74f-5302a66c8ca1 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:55:04.423000 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-94de5065-ca18-4779-a74f-5302a66c8ca1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:04.455560 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 831/1674] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:55:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:55:04.455919 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97308cb5-70f3-42ae-ad02-0342f8184668 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:04.482800 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97308cb5-70f3-42ae-ad02-0342f8184668 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:04.511771 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a275db23-fcc2-4917-a758-d019bd1c4869 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:04.512477 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a275db23-fcc2-4917-a758-d019bd1c4869 admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:04.512832 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a275db23-fcc2-4917-a758-d019bd1c4869 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:04.513211 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a275db23-fcc2-4917-a758-d019bd1c4869 admin admin] PATH_INFO: `/v3/domains` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:04.513954 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a275db23-fcc2-4917-a758-d019bd1c4869 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:55:04.514324 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a275db23-fcc2-4917-a758-d019bd1c4869 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:04.521683 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a275db23-fcc2-4917-a758-d019bd1c4869 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:04.578111 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 844/1675] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Aug 28 00:55:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 119 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:55:04.579660 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94de5065-ca18-4779-a74f-5302a66c8ca1 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:04.594477 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94de5065-ca18-4779-a74f-5302a66c8ca1 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:04.625162 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12fcb078-300e-42ec-8f8f-37be816fe494 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:04.626225 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12fcb078-300e-42ec-8f8f-37be816fe494 admin admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:04.626695 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12fcb078-300e-42ec-8f8f-37be816fe494 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:04.627141 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12fcb078-300e-42ec-8f8f-37be816fe494 admin admin] PATH_INFO: `/v3/projects` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:04.627980 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12fcb078-300e-42ec-8f8f-37be816fe494 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:04.630374 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12fcb078-300e-42ec-8f8f-37be816fe494 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:04.795023 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a275db23-fcc2-4917-a758-d019bd1c4869 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:04.800781 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 832/1676] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 00:55:04 2022] POST /identity/v3/projects => generated 367 bytes in 217 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:55:04.808804 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a275db23-fcc2-4917-a758-d019bd1c4869 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:05.052764 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b1547e2-fe97-48cd-89db-6e4c67739b76 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:05.059150 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b1547e2-fe97-48cd-89db-6e4c67739b76 admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:05.059772 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b1547e2-fe97-48cd-89db-6e4c67739b76 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:05.061821 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b1547e2-fe97-48cd-89db-6e4c67739b76 admin admin] PATH_INFO: `/v3/users` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:05.062774 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b1547e2-fe97-48cd-89db-6e4c67739b76 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:05.074816 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b1547e2-fe97-48cd-89db-6e4c67739b76 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:05.182520 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12fcb078-300e-42ec-8f8f-37be816fe494 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:05.194709 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 845/1677] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 00:55:04 2022] POST /identity/v3/users => generated 284 bytes in 384 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 28 00:55:05.205647 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12fcb078-300e-42ec-8f8f-37be816fe494 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:05.245753 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6d27abb-3736-446d-940c-f62560bd37fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:05.251955 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6d27abb-3736-446d-940c-f62560bd37fb admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:05.254641 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6d27abb-3736-446d-940c-f62560bd37fb admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:05.255023 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6d27abb-3736-446d-940c-f62560bd37fb admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:05.255942 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6d27abb-3736-446d-940c-f62560bd37fb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:05.257632 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6d27abb-3736-446d-940c-f62560bd37fb admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:05.320139 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b1547e2-fe97-48cd-89db-6e4c67739b76 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:05.337378 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 833/1678] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:55:05 2022] GET /identity/v3/roles => generated 1420 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 00:55:05.342891 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b1547e2-fe97-48cd-89db-6e4c67739b76 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:05.362379 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0d6397f-d77e-463d-9513-fdacebfeb7fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:05.363988 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0d6397f-d77e-463d-9513-fdacebfeb7fa admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:05.372059 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0d6397f-d77e-463d-9513-fdacebfeb7fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:05.373667 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0d6397f-d77e-463d-9513-fdacebfeb7fa admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:05.375663 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0d6397f-d77e-463d-9513-fdacebfeb7fa admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:05.380671 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0d6397f-d77e-463d-9513-fdacebfeb7fa admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:05.434365 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 846/1679] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:55:05 2022] GET /identity/v3/roles => generated 1420 bytes in 115 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 00:55:05.438735 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6d27abb-3736-446d-940c-f62560bd37fb admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:05.454701 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6d27abb-3736-446d-940c-f62560bd37fb admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:05.492321 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b13f41dc-c332-410c-b4de-ed05949d5050 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:05.493249 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b13f41dc-c332-410c-b4de-ed05949d5050 admin admin] REQUEST_METHOD: `PUT` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:05.493561 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b13f41dc-c332-410c-b4de-ed05949d5050 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:05.494488 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b13f41dc-c332-410c-b4de-ed05949d5050 admin admin] PATH_INFO: `/v3/projects/1782b3b927be492a9255a0c25d773d34/users/0099f965b9fc4105ab98c565a68bc31c/roles/0487ef7e943e4701915cb8ce0f2899d7` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:05.575008 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b13f41dc-c332-410c-b4de-ed05949d5050 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1782b3b927be492a9255a0c25d773d34, user_id=0099f965b9fc4105ab98c565a68bc31c, role_id=0487ef7e943e4701915cb8ce0f2899d7)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:05.582414 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b13f41dc-c332-410c-b4de-ed05949d5050 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:05.633486 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-830b473c-605f-4256-9f4a-f86eb690f7cf None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:05.634046 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-830b473c-605f-4256-9f4a-f86eb690f7cf None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:05.634550 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-830b473c-605f-4256-9f4a-f86eb690f7cf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:05.645101 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 834/1680] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 00:55:05 2022] PUT /identity/v3/projects/1782b3b927be492a9255a0c25d773d34/users/0099f965b9fc4105ab98c565a68bc31c/roles/0487ef7e943e4701915cb8ce0f2899d7 => generated 0 bytes in 178 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:55:05.725572 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-830b473c-605f-4256-9f4a-f86eb690f7cf None None] MFA Rules not processed for user `0099f965b9fc4105ab98c565a68bc31c`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:55:05.727141 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-830b473c-605f-4256-9f4a-f86eb690f7cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:05.790779 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:55:05.936570 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbf81347-2e7a-48e8-bdc2-b2afaa0cd08f None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:05.936856 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbf81347-2e7a-48e8-bdc2-b2afaa0cd08f None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:05.937160 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbf81347-2e7a-48e8-bdc2-b2afaa0cd08f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:05.984240 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 847/1681] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:55:05 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 353 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:55:06.009060 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-830b473c-605f-4256-9f4a-f86eb690f7cf None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:06.010437 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-830b473c-605f-4256-9f4a-f86eb690f7cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:06.026968 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-bbf81347-2e7a-48e8-bdc2-b2afaa0cd08f None None] MFA Rules not processed for user `4894ef8b630e4bc084799c26096f7c66`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:55:06.028313 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-bbf81347-2e7a-48e8-bdc2-b2afaa0cd08f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:06.296064 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-830b473c-605f-4256-9f4a-f86eb690f7cf None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:06.301203 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 835/1682] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:55:05 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 366 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:55:06.326311 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d79375c5-c378-4398-b28e-ae518fec04ab service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:06.327270 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d79375c5-c378-4398-b28e-ae518fec04ab service cinder] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:06.328034 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d79375c5-c378-4398-b28e-ae518fec04ab service cinder] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:06.328463 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d79375c5-c378-4398-b28e-ae518fec04ab service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:06.355297 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d79375c5-c378-4398-b28e-ae518fec04ab service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:06.357129 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d79375c5-c378-4398-b28e-ae518fec04ab service cinder] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:06.376392 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 848/1683] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:55:06 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 368 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 28 00:55:07.208091 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eba425c-b027-4f2f-be69-f7b5a31224b5 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:07.212194 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eba425c-b027-4f2f-be69-f7b5a31224b5 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:07.213243 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eba425c-b027-4f2f-be69-f7b5a31224b5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:07.298930 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-4eba425c-b027-4f2f-be69-f7b5a31224b5 None None] MFA Rules not processed for user `364a28c108e744728917ba2b0aa2347d`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:55:07.301907 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-4eba425c-b027-4f2f-be69-f7b5a31224b5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:07.533578 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:55:07.625064 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 836/1684] 10.222.0.26 () {58 vars in 1015 bytes} [Sun Aug 28 00:55:07 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 418 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:55:07.859053 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d79375c5-c378-4398-b28e-ae518fec04ab service cinder] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:07.862326 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-d79375c5-c378-4398-b28e-ae518fec04ab service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:07.901164 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d79375c5-c378-4398-b28e-ae518fec04ab service cinder] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:07.913010 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2be164e-c9d0-4790-91b7-9fd26c63af6a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:07.913857 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2be164e-c9d0-4790-91b7-9fd26c63af6a service neutron] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:07.914300 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2be164e-c9d0-4790-91b7-9fd26c63af6a service neutron] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:07.914723 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2be164e-c9d0-4790-91b7-9fd26c63af6a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:07.940190 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2be164e-c9d0-4790-91b7-9fd26c63af6a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:07.945549 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2be164e-c9d0-4790-91b7-9fd26c63af6a service neutron] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:07.974056 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 849/1685] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:55:07 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 116 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:55:10.663079 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-177ea506-2df8-4440-ba43-b85f9d0b3661 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:10.664308 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-177ea506-2df8-4440-ba43-b85f9d0b3661 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:10.665342 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-177ea506-2df8-4440-ba43-b85f9d0b3661 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:10.769235 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-177ea506-2df8-4440-ba43-b85f9d0b3661 None None] MFA Rules not processed for user `3de5d68f67784e33bdcd19a9cdfed744`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:55:10.775516 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-177ea506-2df8-4440-ba43-b85f9d0b3661 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:11.077421 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 837/1686] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:55:10 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 416 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:55:16.332834 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7749d711-fea6-480c-ab02-bcc95ffee6e3 None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:16.333384 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7749d711-fea6-480c-ab02-bcc95ffee6e3 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:16.333459 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7749d711-fea6-480c-ab02-bcc95ffee6e3 None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:16.335216 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 850/1687] 10.222.0.42 () {54 vars in 921 bytes} [Sun Aug 28 00:55:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:55:16.401121 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ee76c05-cad3-4671-81c6-4a87a51846a5 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:16.402211 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ee76c05-cad3-4671-81c6-4a87a51846a5 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:16.402668 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ee76c05-cad3-4671-81c6-4a87a51846a5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:16.445824 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-0ee76c05-cad3-4671-81c6-4a87a51846a5 None None] MFA Rules not processed for user `168b07df50c24593ba73dbde5b037aed`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:55:16.447699 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-0ee76c05-cad3-4671-81c6-4a87a51846a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:16.449766 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7749d711-fea6-480c-ab02-bcc95ffee6e3 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:16.452339 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-7749d711-fea6-480c-ab02-bcc95ffee6e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:16.568090 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 838/1688] 10.222.0.42 () {58 vars in 1018 bytes} [Sun Aug 28 00:55:16 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:55:16.582709 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7749d711-fea6-480c-ab02-bcc95ffee6e3 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:16.587679 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97c73c71-7969-4b77-8627-7feb8fbffe69 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:16.588604 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c73c71-7969-4b77-8627-7feb8fbffe69 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:16.589174 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c73c71-7969-4b77-8627-7feb8fbffe69 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:16.589651 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c73c71-7969-4b77-8627-7feb8fbffe69 admin admin] PATH_INFO: `/v3/users/6365894d1845471e953664344e2a1545` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:16.591366 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97c73c71-7969-4b77-8627-7feb8fbffe69 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6365894d1845471e953664344e2a1545)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:16.594124 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97c73c71-7969-4b77-8627-7feb8fbffe69 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:16.652158 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=93,keystone:DELETE=7 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:55:16.655453 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-97c73c71-7969-4b77-8627-7feb8fbffe69 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6365894d1845471e953664344e2a1545'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:16.658061 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-97c73c71-7969-4b77-8627-7feb8fbffe69 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6365894d1845471e953664344e2a1545'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:16.667762 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-97c73c71-7969-4b77-8627-7feb8fbffe69 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6365894d1845471e953664344e2a1545'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:16.675900 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-97c73c71-7969-4b77-8627-7feb8fbffe69 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6365894d1845471e953664344e2a1545'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:16.677002 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-97c73c71-7969-4b77-8627-7feb8fbffe69 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6365894d1845471e953664344e2a1545'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:16.680811 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3031d4b0-15a6-44e6-ab7f-9d146609f94c None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:16.681157 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3031d4b0-15a6-44e6-ab7f-9d146609f94c None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:16.681388 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3031d4b0-15a6-44e6-ab7f-9d146609f94c None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:16.682908 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 839/1689] 10.222.0.42 () {54 vars in 921 bytes} [Sun Aug 28 00:55:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 28 00:55:16.702872 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 851/1690] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 00:55:16 2022] DELETE /identity/v3/users/6365894d1845471e953664344e2a1545 => generated 0 bytes in 254 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:55:16.707569 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-756c0732-261f-4426-a7a3-85a4ab845e6a None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:16.707852 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-756c0732-261f-4426-a7a3-85a4ab845e6a None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:16.708133 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-756c0732-261f-4426-a7a3-85a4ab845e6a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:16.718775 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9b52331-5465-49d5-9cb6-186045f5d6ea None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:16.719079 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9b52331-5465-49d5-9cb6-186045f5d6ea None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:16.719323 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9b52331-5465-49d5-9cb6-186045f5d6ea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:16.749517 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-f9b52331-5465-49d5-9cb6-186045f5d6ea None None] MFA Rules not processed for user `168b07df50c24593ba73dbde5b037aed`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:55:16.750786 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-f9b52331-5465-49d5-9cb6-186045f5d6ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:16.770447 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-756c0732-261f-4426-a7a3-85a4ab845e6a None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:55:16.771730 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-756c0732-261f-4426-a7a3-85a4ab845e6a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:16.877852 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 840/1691] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:55:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:55:16.888152 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-756c0732-261f-4426-a7a3-85a4ab845e6a None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:16.890009 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-756c0732-261f-4426-a7a3-85a4ab845e6a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:16.901403 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 852/1692] 10.222.0.42 () {58 vars in 1018 bytes} [Sun Aug 28 00:55:16 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:55:16.919975 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:55:16.960026 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-756c0732-261f-4426-a7a3-85a4ab845e6a None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:16.963994 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93b6063f-b8a2-4275-adf5-07eeeca07ed9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:16.965021 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93b6063f-b8a2-4275-adf5-07eeeca07ed9 service neutron] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:16.965712 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93b6063f-b8a2-4275-adf5-07eeeca07ed9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:16.966188 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93b6063f-b8a2-4275-adf5-07eeeca07ed9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:16.978293 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93b6063f-b8a2-4275-adf5-07eeeca07ed9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:16.980344 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93b6063f-b8a2-4275-adf5-07eeeca07ed9 service neutron] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:16.992803 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 841/1693] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:55:16 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:55:17.907610 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9b52331-5465-49d5-9cb6-186045f5d6ea None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:17.911515 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-f9b52331-5465-49d5-9cb6-186045f5d6ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:17.961929 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9b52331-5465-49d5-9cb6-186045f5d6ea None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:17.970397 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44a401bd-5e1a-4c98-a81b-17ae0da3141a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:17.971347 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44a401bd-5e1a-4c98-a81b-17ae0da3141a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:17.971808 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44a401bd-5e1a-4c98-a81b-17ae0da3141a admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:17.972277 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44a401bd-5e1a-4c98-a81b-17ae0da3141a admin admin] PATH_INFO: `/v3/projects/02fc0fc894b641bfa6b5d5e2c66a7ee4` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:17.974557 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44a401bd-5e1a-4c98-a81b-17ae0da3141a admin admin] RBAC: Authorizing `identity:delete_project(project_id=02fc0fc894b641bfa6b5d5e2c66a7ee4)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:17.976710 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44a401bd-5e1a-4c98-a81b-17ae0da3141a admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:18.063324 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-44a401bd-5e1a-4c98-a81b-17ae0da3141a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '02fc0fc894b641bfa6b5d5e2c66a7ee4', 'request_id': 'req-44a401bd-5e1a-4c98-a81b-17ae0da3141a'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:18.079535 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-44a401bd-5e1a-4c98-a81b-17ae0da3141a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '02fc0fc894b641bfa6b5d5e2c66a7ee4', 'request_id': 'req-44a401bd-5e1a-4c98-a81b-17ae0da3141a'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:18.091419 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93b6063f-b8a2-4275-adf5-07eeeca07ed9 service neutron] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:18.093370 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-93b6063f-b8a2-4275-adf5-07eeeca07ed9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:18.107854 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-44a401bd-5e1a-4c98-a81b-17ae0da3141a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '02fc0fc894b641bfa6b5d5e2c66a7ee4', 'request_id': 'req-44a401bd-5e1a-4c98-a81b-17ae0da3141a'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:18.132292 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93b6063f-b8a2-4275-adf5-07eeeca07ed9 service neutron] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:18.134655 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-44a401bd-5e1a-4c98-a81b-17ae0da3141a admin admin] The token cache is being invalidate because project 02fc0fc894b641bfa6b5d5e2c66a7ee4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81508) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 00:55:18.135087 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-44a401bd-5e1a-4c98-a81b-17ae0da3141a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:18.136618 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-44a401bd-5e1a-4c98-a81b-17ae0da3141a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:18.138830 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 853/1694] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 00:55:17 2022] DELETE /identity/v3/projects/02fc0fc894b641bfa6b5d5e2c66a7ee4 => generated 0 bytes in 232 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:55:18.143067 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44a401bd-5e1a-4c98-a81b-17ae0da3141a admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:18.144784 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-44a401bd-5e1a-4c98-a81b-17ae0da3141a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:18.203782 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:55:18.231520 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44a401bd-5e1a-4c98-a81b-17ae0da3141a admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:18.236932 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6654569f-48c2-437e-9dca-2aaadfa8caf7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:18.237692 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6654569f-48c2-437e-9dca-2aaadfa8caf7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:18.238053 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6654569f-48c2-437e-9dca-2aaadfa8caf7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:18.238546 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6654569f-48c2-437e-9dca-2aaadfa8caf7 admin admin] PATH_INFO: `/v3/users/4a475e733cd24983a2a3cecbe7197a0b` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:18.240549 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6654569f-48c2-437e-9dca-2aaadfa8caf7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4a475e733cd24983a2a3cecbe7197a0b)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:18.242616 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6654569f-48c2-437e-9dca-2aaadfa8caf7 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:18.320657 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-6654569f-48c2-437e-9dca-2aaadfa8caf7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4a475e733cd24983a2a3cecbe7197a0b'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:18.321687 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-6654569f-48c2-437e-9dca-2aaadfa8caf7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4a475e733cd24983a2a3cecbe7197a0b'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:18.337082 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-6654569f-48c2-437e-9dca-2aaadfa8caf7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4a475e733cd24983a2a3cecbe7197a0b'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:18.347396 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e8f19d2-e7b1-4eb7-8845-4d48b4af88b9 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '168b07df50c24593ba73dbde5b037aed', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:18.348093 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e8f19d2-e7b1-4eb7-8845-4d48b4af88b9 service nova] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:18.348424 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e8f19d2-e7b1-4eb7-8845-4d48b4af88b9 service nova] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:18.348702 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e8f19d2-e7b1-4eb7-8845-4d48b4af88b9 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:18.351714 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-9e8f19d2-e7b1-4eb7-8845-4d48b4af88b9 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:18.358650 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-6654569f-48c2-437e-9dca-2aaadfa8caf7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4a475e733cd24983a2a3cecbe7197a0b'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:18.361561 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-6654569f-48c2-437e-9dca-2aaadfa8caf7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4a475e733cd24983a2a3cecbe7197a0b'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:18.382080 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e8f19d2-e7b1-4eb7-8845-4d48b4af88b9 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:18.387369 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e8f19d2-e7b1-4eb7-8845-4d48b4af88b9 service nova] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:18.418337 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 842/1695] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:55:18 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 326 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:55:18.443254 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 854/1696] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 00:55:18 2022] DELETE /identity/v3/users/4a475e733cd24983a2a3cecbe7197a0b => generated 0 bytes in 297 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 28 00:55:18.463728 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e8f19d2-e7b1-4eb7-8845-4d48b4af88b9 service nova] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:18.465665 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-9e8f19d2-e7b1-4eb7-8845-4d48b4af88b9 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:18.492346 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e8f19d2-e7b1-4eb7-8845-4d48b4af88b9 service nova] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:18.683487 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24c1800e-fe4d-43a6-98ec-748bf22e2db2 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '168b07df50c24593ba73dbde5b037aed', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:18.684189 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24c1800e-fe4d-43a6-98ec-748bf22e2db2 service nova] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:18.684546 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24c1800e-fe4d-43a6-98ec-748bf22e2db2 service nova] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:18.684732 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24c1800e-fe4d-43a6-98ec-748bf22e2db2 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:18.686850 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-24c1800e-fe4d-43a6-98ec-748bf22e2db2 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:18.707275 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24c1800e-fe4d-43a6-98ec-748bf22e2db2 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:18.710423 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24c1800e-fe4d-43a6-98ec-748bf22e2db2 service nova] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:18.731161 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 843/1697] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:55:18 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 268 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:55:18.988311 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6654569f-48c2-437e-9dca-2aaadfa8caf7 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:18.990023 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-6654569f-48c2-437e-9dca-2aaadfa8caf7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:19.073115 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6654569f-48c2-437e-9dca-2aaadfa8caf7 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:19.081514 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a27eeb90-0bef-4a70-9520-b2aa61c3670d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:19.082554 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a27eeb90-0bef-4a70-9520-b2aa61c3670d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:19.082881 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a27eeb90-0bef-4a70-9520-b2aa61c3670d admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:19.083153 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a27eeb90-0bef-4a70-9520-b2aa61c3670d admin admin] PATH_INFO: `/v3/projects/5c6f87af024943ee81d82cded504fc67` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:19.084889 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a27eeb90-0bef-4a70-9520-b2aa61c3670d admin admin] RBAC: Authorizing `identity:delete_project(project_id=5c6f87af024943ee81d82cded504fc67)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:19.087283 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a27eeb90-0bef-4a70-9520-b2aa61c3670d admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:19.155432 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-a27eeb90-0bef-4a70-9520-b2aa61c3670d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5c6f87af024943ee81d82cded504fc67', 'request_id': 'req-a27eeb90-0bef-4a70-9520-b2aa61c3670d'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:19.163237 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-a27eeb90-0bef-4a70-9520-b2aa61c3670d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5c6f87af024943ee81d82cded504fc67', 'request_id': 'req-a27eeb90-0bef-4a70-9520-b2aa61c3670d'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:19.165637 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=83,keystone:DELETE=16,keystone:INSERT=1 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:55:19.174376 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-a27eeb90-0bef-4a70-9520-b2aa61c3670d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5c6f87af024943ee81d82cded504fc67', 'request_id': 'req-a27eeb90-0bef-4a70-9520-b2aa61c3670d'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:19.193418 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-a27eeb90-0bef-4a70-9520-b2aa61c3670d admin admin] The token cache is being invalidate because project 5c6f87af024943ee81d82cded504fc67 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81508) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 00:55:19.193869 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-a27eeb90-0bef-4a70-9520-b2aa61c3670d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:19.194804 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-a27eeb90-0bef-4a70-9520-b2aa61c3670d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:19.196580 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 855/1698] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 00:55:18 2022] DELETE /identity/v3/projects/5c6f87af024943ee81d82cded504fc67 => generated 0 bytes in 210 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:55:19.201824 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-695b8dd3-8edc-4dc6-8186-2a9c0b015b29 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:19.202346 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-695b8dd3-8edc-4dc6-8186-2a9c0b015b29 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:19.202587 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-695b8dd3-8edc-4dc6-8186-2a9c0b015b29 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:19.224854 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-695b8dd3-8edc-4dc6-8186-2a9c0b015b29 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:55:19.226759 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-695b8dd3-8edc-4dc6-8186-2a9c0b015b29 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:19.267374 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:55:19.307904 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 844/1699] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:55:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:55:19.315878 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c5b9045-75ea-434b-aed0-cebaf05fd214 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:19.316170 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c5b9045-75ea-434b-aed0-cebaf05fd214 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:19.316508 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c5b9045-75ea-434b-aed0-cebaf05fd214 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:19.345879 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-7c5b9045-75ea-434b-aed0-cebaf05fd214 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:55:19.347258 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-7c5b9045-75ea-434b-aed0-cebaf05fd214 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:19.357119 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 856/1700] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:55:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:55:19.361559 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-695b8dd3-8edc-4dc6-8186-2a9c0b015b29 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:19.372058 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-695b8dd3-8edc-4dc6-8186-2a9c0b015b29 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:19.377264 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e57187de-6348-49cb-b7dc-5a0440168865 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:19.378077 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e57187de-6348-49cb-b7dc-5a0440168865 admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:19.378361 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e57187de-6348-49cb-b7dc-5a0440168865 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:19.378639 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e57187de-6348-49cb-b7dc-5a0440168865 admin admin] PATH_INFO: `/v3/domains` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:19.379293 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e57187de-6348-49cb-b7dc-5a0440168865 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:55:19.379596 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e57187de-6348-49cb-b7dc-5a0440168865 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:19.381241 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e57187de-6348-49cb-b7dc-5a0440168865 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:19.393971 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 845/1701] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Aug 28 00:55:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:55:19.398553 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c5b9045-75ea-434b-aed0-cebaf05fd214 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:19.407885 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c5b9045-75ea-434b-aed0-cebaf05fd214 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:19.426218 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a54569dc-9652-49e0-9aec-3c0de32acbf6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:19.427182 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a54569dc-9652-49e0-9aec-3c0de32acbf6 admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:19.427783 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a54569dc-9652-49e0-9aec-3c0de32acbf6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:19.428381 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a54569dc-9652-49e0-9aec-3c0de32acbf6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:19.429276 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a54569dc-9652-49e0-9aec-3c0de32acbf6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:19.432616 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a54569dc-9652-49e0-9aec-3c0de32acbf6 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:19.493299 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 857/1702] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 00:55:19 2022] POST /identity/v3/projects => generated 379 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:55:19.494558 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e57187de-6348-49cb-b7dc-5a0440168865 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:19.510352 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e57187de-6348-49cb-b7dc-5a0440168865 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:19.599026 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7387af8-9be4-4396-9f63-decca1e5b3bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:19.599849 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7387af8-9be4-4396-9f63-decca1e5b3bb admin admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:19.600277 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7387af8-9be4-4396-9f63-decca1e5b3bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:19.600594 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7387af8-9be4-4396-9f63-decca1e5b3bb admin admin] PATH_INFO: `/v3/users` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:19.601478 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7387af8-9be4-4396-9f63-decca1e5b3bb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:19.603461 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7387af8-9be4-4396-9f63-decca1e5b3bb admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:19.628110 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 846/1703] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 00:55:19 2022] POST /identity/v3/users => generated 290 bytes in 134 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 28 00:55:19.632250 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a54569dc-9652-49e0-9aec-3c0de32acbf6 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:19.645676 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a54569dc-9652-49e0-9aec-3c0de32acbf6 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:19.664813 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c49e3734-7e99-4352-9cac-3ccff2ed7c47 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:19.666552 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c49e3734-7e99-4352-9cac-3ccff2ed7c47 admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:19.666907 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c49e3734-7e99-4352-9cac-3ccff2ed7c47 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:19.667294 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c49e3734-7e99-4352-9cac-3ccff2ed7c47 admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:19.668702 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c49e3734-7e99-4352-9cac-3ccff2ed7c47 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:19.670728 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c49e3734-7e99-4352-9cac-3ccff2ed7c47 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:19.691515 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 858/1704] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:55:19 2022] GET /identity/v3/roles => generated 1420 bytes in 60 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 28 00:55:19.695162 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7387af8-9be4-4396-9f63-decca1e5b3bb admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:19.715751 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7387af8-9be4-4396-9f63-decca1e5b3bb admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:19.721363 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ccd8b23-00fe-4dcb-a954-28f22992f911 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:19.722297 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ccd8b23-00fe-4dcb-a954-28f22992f911 admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:19.722618 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ccd8b23-00fe-4dcb-a954-28f22992f911 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:19.722889 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ccd8b23-00fe-4dcb-a954-28f22992f911 admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:19.723564 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ccd8b23-00fe-4dcb-a954-28f22992f911 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:19.725214 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ccd8b23-00fe-4dcb-a954-28f22992f911 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:19.738695 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 847/1705] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:55:19 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 00:55:19.743521 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c49e3734-7e99-4352-9cac-3ccff2ed7c47 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:19.754403 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c49e3734-7e99-4352-9cac-3ccff2ed7c47 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:19.759793 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-436187d1-f72c-4d53-8e04-9694199db039 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:19.760463 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-436187d1-f72c-4d53-8e04-9694199db039 admin admin] REQUEST_METHOD: `PUT` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:19.760745 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-436187d1-f72c-4d53-8e04-9694199db039 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:19.761075 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-436187d1-f72c-4d53-8e04-9694199db039 admin admin] PATH_INFO: `/v3/projects/8898528286db4e8a8af4288c059a3b5c/users/6e268fa0024947319a3b736641c6e0b6/roles/0487ef7e943e4701915cb8ce0f2899d7` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:19.789179 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-436187d1-f72c-4d53-8e04-9694199db039 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8898528286db4e8a8af4288c059a3b5c, user_id=6e268fa0024947319a3b736641c6e0b6, role_id=0487ef7e943e4701915cb8ce0f2899d7)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:19.792069 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-436187d1-f72c-4d53-8e04-9694199db039 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:19.813025 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 859/1706] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 00:55:19 2022] PUT /identity/v3/projects/8898528286db4e8a8af4288c059a3b5c/users/6e268fa0024947319a3b736641c6e0b6/roles/0487ef7e943e4701915cb8ce0f2899d7 => generated 0 bytes in 66 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 28 00:55:19.822917 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-829681bb-28a8-4100-aee9-806d99c0b724 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:19.823437 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-829681bb-28a8-4100-aee9-806d99c0b724 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:19.823723 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-829681bb-28a8-4100-aee9-806d99c0b724 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:19.861743 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-829681bb-28a8-4100-aee9-806d99c0b724 None None] MFA Rules not processed for user `6e268fa0024947319a3b736641c6e0b6`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:55:19.862876 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-829681bb-28a8-4100-aee9-806d99c0b724 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:19.938963 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 848/1707] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:55:19 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:55:19.943525 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-436187d1-f72c-4d53-8e04-9694199db039 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:20.021937 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-436187d1-f72c-4d53-8e04-9694199db039 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:20.027269 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0716e5a8-44ca-4cbb-a0ea-316e6ffb1cf7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:20.028180 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0716e5a8-44ca-4cbb-a0ea-316e6ffb1cf7 admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:20.028533 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0716e5a8-44ca-4cbb-a0ea-316e6ffb1cf7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:20.028872 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0716e5a8-44ca-4cbb-a0ea-316e6ffb1cf7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:20.029620 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0716e5a8-44ca-4cbb-a0ea-316e6ffb1cf7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:20.031641 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0716e5a8-44ca-4cbb-a0ea-316e6ffb1cf7 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:20.067830 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 860/1708] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 00:55:19 2022] POST /identity/v3/projects => generated 379 bytes in 125 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:55:20.071895 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-829681bb-28a8-4100-aee9-806d99c0b724 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:20.085661 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-829681bb-28a8-4100-aee9-806d99c0b724 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:20.132571 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:55:20.180979 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4fb8174-b1d7-4806-9e67-f06aa37849ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:20.183869 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4fb8174-b1d7-4806-9e67-f06aa37849ca admin admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:20.184155 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4fb8174-b1d7-4806-9e67-f06aa37849ca admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:20.184426 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4fb8174-b1d7-4806-9e67-f06aa37849ca admin admin] PATH_INFO: `/v3/users` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:20.185345 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4fb8174-b1d7-4806-9e67-f06aa37849ca admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:20.187314 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4fb8174-b1d7-4806-9e67-f06aa37849ca admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:20.213682 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 849/1709] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 00:55:20 2022] POST /identity/v3/users => generated 290 bytes in 142 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 28 00:55:20.218029 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0716e5a8-44ca-4cbb-a0ea-316e6ffb1cf7 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:20.229260 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0716e5a8-44ca-4cbb-a0ea-316e6ffb1cf7 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:20.233976 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb5a6dde-933d-4325-a4a2-4e64d990d941 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:20.234579 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb5a6dde-933d-4325-a4a2-4e64d990d941 admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:20.234880 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb5a6dde-933d-4325-a4a2-4e64d990d941 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:20.235146 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb5a6dde-933d-4325-a4a2-4e64d990d941 admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:20.235817 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb5a6dde-933d-4325-a4a2-4e64d990d941 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:20.237489 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb5a6dde-933d-4325-a4a2-4e64d990d941 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:20.248415 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 861/1710] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:55:20 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 00:55:20.252544 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4fb8174-b1d7-4806-9e67-f06aa37849ca admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:20.263853 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4fb8174-b1d7-4806-9e67-f06aa37849ca admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:20.268431 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dd3bba5-6888-4776-985d-8c2a256b2cab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:20.269068 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dd3bba5-6888-4776-985d-8c2a256b2cab admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:20.269347 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dd3bba5-6888-4776-985d-8c2a256b2cab admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:20.269620 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dd3bba5-6888-4776-985d-8c2a256b2cab admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:20.270261 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dd3bba5-6888-4776-985d-8c2a256b2cab admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:20.271902 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dd3bba5-6888-4776-985d-8c2a256b2cab admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:20.282637 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 850/1711] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:55:20 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 00:55:20.287070 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb5a6dde-933d-4325-a4a2-4e64d990d941 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:20.297391 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb5a6dde-933d-4325-a4a2-4e64d990d941 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:20.302024 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67066803-2c5e-4d2f-bc0c-4e72aca1a4d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:20.302567 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67066803-2c5e-4d2f-bc0c-4e72aca1a4d4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:20.302758 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67066803-2c5e-4d2f-bc0c-4e72aca1a4d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:20.303050 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67066803-2c5e-4d2f-bc0c-4e72aca1a4d4 admin admin] PATH_INFO: `/v3/projects/24a84ac76bb84719a5c6110fc4015f09/users/6a26d8fbca8a411197222907abcb54b5/roles/0487ef7e943e4701915cb8ce0f2899d7` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:20.326444 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67066803-2c5e-4d2f-bc0c-4e72aca1a4d4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=24a84ac76bb84719a5c6110fc4015f09, user_id=6a26d8fbca8a411197222907abcb54b5, role_id=0487ef7e943e4701915cb8ce0f2899d7)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:20.329638 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67066803-2c5e-4d2f-bc0c-4e72aca1a4d4 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:20.347783 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 862/1712] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 00:55:20 2022] PUT /identity/v3/projects/24a84ac76bb84719a5c6110fc4015f09/users/6a26d8fbca8a411197222907abcb54b5/roles/0487ef7e943e4701915cb8ce0f2899d7 => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 28 00:55:20.361627 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0da8f281-4a4f-4398-b938-f976da269471 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:20.361917 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0da8f281-4a4f-4398-b938-f976da269471 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:20.362443 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0da8f281-4a4f-4398-b938-f976da269471 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:20.405436 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-0da8f281-4a4f-4398-b938-f976da269471 None None] MFA Rules not processed for user `6a26d8fbca8a411197222907abcb54b5`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:55:20.406300 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-0da8f281-4a4f-4398-b938-f976da269471 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:20.486550 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 851/1713] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:55:20 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:55:20.492321 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67066803-2c5e-4d2f-bc0c-4e72aca1a4d4 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:20.546193 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=94,keystone:INSERT=5,keystone:DELETE=1 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:55:20.598376 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67066803-2c5e-4d2f-bc0c-4e72aca1a4d4 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:20.604255 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3782f7e1-8f27-4dc5-8153-25e2a53fd287 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:20.605084 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3782f7e1-8f27-4dc5-8153-25e2a53fd287 admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:20.605416 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3782f7e1-8f27-4dc5-8153-25e2a53fd287 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:20.605688 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3782f7e1-8f27-4dc5-8153-25e2a53fd287 admin admin] PATH_INFO: `/v3/projects` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:20.606450 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3782f7e1-8f27-4dc5-8153-25e2a53fd287 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:20.608600 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3782f7e1-8f27-4dc5-8153-25e2a53fd287 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:20.645754 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 863/1714] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 00:55:20 2022] POST /identity/v3/projects => generated 377 bytes in 151 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:55:20.646972 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0da8f281-4a4f-4398-b938-f976da269471 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:20.659911 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0da8f281-4a4f-4398-b938-f976da269471 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:20.745286 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b7ad7a0-748a-4f06-bbc2-02cb075baecd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:20.745925 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b7ad7a0-748a-4f06-bbc2-02cb075baecd admin admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:20.746263 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b7ad7a0-748a-4f06-bbc2-02cb075baecd admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:20.746600 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b7ad7a0-748a-4f06-bbc2-02cb075baecd admin admin] PATH_INFO: `/v3/users` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:20.747440 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b7ad7a0-748a-4f06-bbc2-02cb075baecd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:20.749595 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b7ad7a0-748a-4f06-bbc2-02cb075baecd admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:20.776600 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 852/1715] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 00:55:20 2022] POST /identity/v3/users => generated 295 bytes in 130 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:55:20.780426 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3782f7e1-8f27-4dc5-8153-25e2a53fd287 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:20.791378 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3782f7e1-8f27-4dc5-8153-25e2a53fd287 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:20.797350 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d606c0d-6af8-4d0f-9efc-18f8dbea37ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:20.798004 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d606c0d-6af8-4d0f-9efc-18f8dbea37ae admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:20.798329 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d606c0d-6af8-4d0f-9efc-18f8dbea37ae admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:20.798607 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d606c0d-6af8-4d0f-9efc-18f8dbea37ae admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:20.799235 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d606c0d-6af8-4d0f-9efc-18f8dbea37ae admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:20.800832 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d606c0d-6af8-4d0f-9efc-18f8dbea37ae admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:20.813745 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 864/1716] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:55:20 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 00:55:20.820212 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b7ad7a0-748a-4f06-bbc2-02cb075baecd admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:20.833177 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b7ad7a0-748a-4f06-bbc2-02cb075baecd admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:20.838361 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-785784cc-c197-4373-9622-137fab9e331a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:20.839111 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-785784cc-c197-4373-9622-137fab9e331a admin admin] REQUEST_METHOD: `PUT` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:20.839454 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-785784cc-c197-4373-9622-137fab9e331a admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:20.839882 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-785784cc-c197-4373-9622-137fab9e331a admin admin] PATH_INFO: `/v3/projects/ef293af3c6344f448140cf090632af81/users/80139101c3154069a094e7cb56bcc565/roles/f08d6043261747e3b4763c744eef7dfb` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:20.857506 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-785784cc-c197-4373-9622-137fab9e331a admin admin] RBAC: Authorizing `identity:create_grant(project_id=ef293af3c6344f448140cf090632af81, user_id=80139101c3154069a094e7cb56bcc565, role_id=f08d6043261747e3b4763c744eef7dfb)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:20.860636 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-785784cc-c197-4373-9622-137fab9e331a admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:20.880023 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 853/1717] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 00:55:20 2022] PUT /identity/v3/projects/ef293af3c6344f448140cf090632af81/users/80139101c3154069a094e7cb56bcc565/roles/f08d6043261747e3b4763c744eef7dfb => generated 0 bytes in 62 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:55:20.883890 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d606c0d-6af8-4d0f-9efc-18f8dbea37ae admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:20.895857 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d606c0d-6af8-4d0f-9efc-18f8dbea37ae admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:20.961522 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67c629b2-999b-4aa2-b0a5-fcd8966dbf43 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:20.962136 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c629b2-999b-4aa2-b0a5-fcd8966dbf43 admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:20.962413 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c629b2-999b-4aa2-b0a5-fcd8966dbf43 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:20.962681 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c629b2-999b-4aa2-b0a5-fcd8966dbf43 admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:20.963321 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67c629b2-999b-4aa2-b0a5-fcd8966dbf43 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:20.964912 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67c629b2-999b-4aa2-b0a5-fcd8966dbf43 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:20.975509 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 865/1718] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:55:20 2022] GET /identity/v3/roles => generated 1420 bytes in 92 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 00:55:20.980019 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-785784cc-c197-4373-9622-137fab9e331a admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:20.991199 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-785784cc-c197-4373-9622-137fab9e331a admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:20.996200 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3dad4bb-1470-491b-aa31-36797c096f99 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:20.997020 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3dad4bb-1470-491b-aa31-36797c096f99 admin admin] REQUEST_METHOD: `PUT` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:20.997406 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3dad4bb-1470-491b-aa31-36797c096f99 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:20.997701 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3dad4bb-1470-491b-aa31-36797c096f99 admin admin] PATH_INFO: `/v3/projects/ef293af3c6344f448140cf090632af81/users/80139101c3154069a094e7cb56bcc565/roles/f08d6043261747e3b4763c744eef7dfb` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:20.999636 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3dad4bb-1470-491b-aa31-36797c096f99 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ef293af3c6344f448140cf090632af81, user_id=80139101c3154069a094e7cb56bcc565, role_id=f08d6043261747e3b4763c744eef7dfb)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:21.002996 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3dad4bb-1470-491b-aa31-36797c096f99 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:21.005585 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:55:21.017990 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 854/1719] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 00:55:20 2022] PUT /identity/v3/projects/ef293af3c6344f448140cf090632af81/users/80139101c3154069a094e7cb56bcc565/roles/f08d6043261747e3b4763c744eef7dfb => generated 0 bytes in 39 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:55:21.028808 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c356f2b-b82e-4c51-b7da-34e322083c55 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:21.029295 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c356f2b-b82e-4c51-b7da-34e322083c55 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:21.029397 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c356f2b-b82e-4c51-b7da-34e322083c55 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:21.064229 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-5c356f2b-b82e-4c51-b7da-34e322083c55 None None] MFA Rules not processed for user `80139101c3154069a094e7cb56bcc565`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:55:21.065485 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-5c356f2b-b82e-4c51-b7da-34e322083c55 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:21.141401 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 866/1720] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:55:21 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:55:21.153151 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3dad4bb-1470-491b-aa31-36797c096f99 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:21.154962 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-b3dad4bb-1470-491b-aa31-36797c096f99 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:21.219923 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3dad4bb-1470-491b-aa31-36797c096f99 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:21.223785 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fefe766-d74a-40bf-a8a4-321371f7327f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:21.224405 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fefe766-d74a-40bf-a8a4-321371f7327f service cinder] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:21.224683 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fefe766-d74a-40bf-a8a4-321371f7327f service cinder] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:21.224976 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fefe766-d74a-40bf-a8a4-321371f7327f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:21.234576 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fefe766-d74a-40bf-a8a4-321371f7327f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:21.236070 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fefe766-d74a-40bf-a8a4-321371f7327f service cinder] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:21.247071 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 855/1721] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:55:21 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:55:21.289940 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c356f2b-b82e-4c51-b7da-34e322083c55 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:21.291294 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-5c356f2b-b82e-4c51-b7da-34e322083c55 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:21.303637 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c356f2b-b82e-4c51-b7da-34e322083c55 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:21.307165 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2de863b9-7d13-478c-aa63-6c670ddaf9b3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:21.307791 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2de863b9-7d13-478c-aa63-6c670ddaf9b3 service cinder] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:21.308048 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2de863b9-7d13-478c-aa63-6c670ddaf9b3 service cinder] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:21.308326 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2de863b9-7d13-478c-aa63-6c670ddaf9b3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:21.336529 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:55:21.368747 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2de863b9-7d13-478c-aa63-6c670ddaf9b3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:21.370229 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2de863b9-7d13-478c-aa63-6c670ddaf9b3 service cinder] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:21.379162 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 867/1722] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:55:21 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:55:28.638206 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fefe766-d74a-40bf-a8a4-321371f7327f service cinder] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:28.734838 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fefe766-d74a-40bf-a8a4-321371f7327f service cinder] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:28.747268 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbbfa1b9-5c55-4b1e-97fa-ba48c61627dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:28.747948 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbbfa1b9-5c55-4b1e-97fa-ba48c61627dd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:28.748227 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbbfa1b9-5c55-4b1e-97fa-ba48c61627dd admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:28.748492 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbbfa1b9-5c55-4b1e-97fa-ba48c61627dd admin admin] PATH_INFO: `/v3/users/6e268fa0024947319a3b736641c6e0b6` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:28.749654 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbbfa1b9-5c55-4b1e-97fa-ba48c61627dd admin admin] RBAC: Authorizing `identity:delete_user(user_id=6e268fa0024947319a3b736641c6e0b6)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:28.751685 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbbfa1b9-5c55-4b1e-97fa-ba48c61627dd admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:28.811665 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-bbbfa1b9-5c55-4b1e-97fa-ba48c61627dd admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6e268fa0024947319a3b736641c6e0b6'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:28.826851 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-bbbfa1b9-5c55-4b1e-97fa-ba48c61627dd admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6e268fa0024947319a3b736641c6e0b6'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:28.837770 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-bbbfa1b9-5c55-4b1e-97fa-ba48c61627dd admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6e268fa0024947319a3b736641c6e0b6'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:28.846998 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-bbbfa1b9-5c55-4b1e-97fa-ba48c61627dd admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6e268fa0024947319a3b736641c6e0b6'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:28.849391 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-bbbfa1b9-5c55-4b1e-97fa-ba48c61627dd admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6e268fa0024947319a3b736641c6e0b6'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:28.857023 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 856/1723] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 00:55:28 2022] DELETE /identity/v3/users/6e268fa0024947319a3b736641c6e0b6 => generated 0 bytes in 219 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:55:28.861641 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87ae5dd8-445e-447c-b94d-a6255e6db407 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:28.862212 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87ae5dd8-445e-447c-b94d-a6255e6db407 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:28.862495 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87ae5dd8-445e-447c-b94d-a6255e6db407 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:28.887848 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-87ae5dd8-445e-447c-b94d-a6255e6db407 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:55:28.889168 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-87ae5dd8-445e-447c-b94d-a6255e6db407 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:28.977110 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 868/1724] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:55:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:55:28.986439 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbbfa1b9-5c55-4b1e-97fa-ba48c61627dd admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:28.988201 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-bbbfa1b9-5c55-4b1e-97fa-ba48c61627dd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:29.018432 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:INSERT=2,keystone:SELECT=88,keystone:DELETE=10 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:55:29.096401 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbbfa1b9-5c55-4b1e-97fa-ba48c61627dd admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:29.106747 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-402c38db-5e12-4aa1-9e42-caa8794cc507 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:29.118051 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-402c38db-5e12-4aa1-9e42-caa8794cc507 service neutron] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:29.118346 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-402c38db-5e12-4aa1-9e42-caa8794cc507 service neutron] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:29.118579 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-402c38db-5e12-4aa1-9e42-caa8794cc507 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:29.131685 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-402c38db-5e12-4aa1-9e42-caa8794cc507 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:29.133316 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-402c38db-5e12-4aa1-9e42-caa8794cc507 service neutron] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:29.147283 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 857/1725] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:55:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 161 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:55:29.780883 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87ae5dd8-445e-447c-b94d-a6255e6db407 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:29.782613 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-87ae5dd8-445e-447c-b94d-a6255e6db407 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:29.797920 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87ae5dd8-445e-447c-b94d-a6255e6db407 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:29.803141 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e15f14f-d177-4dcb-9af5-5bdf2494c99a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:29.804042 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e15f14f-d177-4dcb-9af5-5bdf2494c99a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:29.804365 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e15f14f-d177-4dcb-9af5-5bdf2494c99a admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:29.804641 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e15f14f-d177-4dcb-9af5-5bdf2494c99a admin admin] PATH_INFO: `/v3/projects/8898528286db4e8a8af4288c059a3b5c` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:29.806004 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e15f14f-d177-4dcb-9af5-5bdf2494c99a admin admin] RBAC: Authorizing `identity:delete_project(project_id=8898528286db4e8a8af4288c059a3b5c)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:29.808121 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e15f14f-d177-4dcb-9af5-5bdf2494c99a admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:29.887678 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-8e15f14f-d177-4dcb-9af5-5bdf2494c99a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8898528286db4e8a8af4288c059a3b5c', 'request_id': 'req-8e15f14f-d177-4dcb-9af5-5bdf2494c99a'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:29.894514 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-8e15f14f-d177-4dcb-9af5-5bdf2494c99a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8898528286db4e8a8af4288c059a3b5c', 'request_id': 'req-8e15f14f-d177-4dcb-9af5-5bdf2494c99a'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:29.903173 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-8e15f14f-d177-4dcb-9af5-5bdf2494c99a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8898528286db4e8a8af4288c059a3b5c', 'request_id': 'req-8e15f14f-d177-4dcb-9af5-5bdf2494c99a'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:29.926529 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-8e15f14f-d177-4dcb-9af5-5bdf2494c99a admin admin] The token cache is being invalidate because project 8898528286db4e8a8af4288c059a3b5c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81508) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 00:55:29.926818 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-8e15f14f-d177-4dcb-9af5-5bdf2494c99a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:29.927466 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-8e15f14f-d177-4dcb-9af5-5bdf2494c99a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:29.929317 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 869/1726] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 00:55:29 2022] DELETE /identity/v3/projects/8898528286db4e8a8af4288c059a3b5c => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:55:29.933635 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-402c38db-5e12-4aa1-9e42-caa8794cc507 service neutron] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:29.946679 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-402c38db-5e12-4aa1-9e42-caa8794cc507 service neutron] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:30.034471 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f209c9b-893d-4339-8548-66fecaad033d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:30.035117 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f209c9b-893d-4339-8548-66fecaad033d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:30.035393 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f209c9b-893d-4339-8548-66fecaad033d admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:30.035690 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f209c9b-893d-4339-8548-66fecaad033d admin admin] PATH_INFO: `/v3/users/6a26d8fbca8a411197222907abcb54b5` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:30.036839 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f209c9b-893d-4339-8548-66fecaad033d admin admin] RBAC: Authorizing `identity:delete_user(user_id=6a26d8fbca8a411197222907abcb54b5)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:30.038802 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f209c9b-893d-4339-8548-66fecaad033d admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:30.094512 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-1f209c9b-893d-4339-8548-66fecaad033d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6a26d8fbca8a411197222907abcb54b5'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:30.114196 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-1f209c9b-893d-4339-8548-66fecaad033d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6a26d8fbca8a411197222907abcb54b5'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:30.136775 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-1f209c9b-893d-4339-8548-66fecaad033d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6a26d8fbca8a411197222907abcb54b5'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:30.141610 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e15f14f-d177-4dcb-9af5-5bdf2494c99a admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:30.144889 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-8e15f14f-d177-4dcb-9af5-5bdf2494c99a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:30.149082 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-1f209c9b-893d-4339-8548-66fecaad033d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6a26d8fbca8a411197222907abcb54b5'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:30.149797 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-1f209c9b-893d-4339-8548-66fecaad033d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6a26d8fbca8a411197222907abcb54b5'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:30.161530 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 858/1727] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 00:55:29 2022] DELETE /identity/v3/users/6a26d8fbca8a411197222907abcb54b5 => generated 0 bytes in 225 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:55:30.163113 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e15f14f-d177-4dcb-9af5-5bdf2494c99a admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:30.235972 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:55:30.252448 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f4a48b3-66d8-410d-b0af-da69157a72b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:30.253184 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f4a48b3-66d8-410d-b0af-da69157a72b1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:30.253535 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f4a48b3-66d8-410d-b0af-da69157a72b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:30.253852 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f4a48b3-66d8-410d-b0af-da69157a72b1 admin admin] PATH_INFO: `/v3/users/0099f965b9fc4105ab98c565a68bc31c` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:30.255033 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f4a48b3-66d8-410d-b0af-da69157a72b1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0099f965b9fc4105ab98c565a68bc31c)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:30.257211 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f4a48b3-66d8-410d-b0af-da69157a72b1 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:30.314611 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-3f4a48b3-66d8-410d-b0af-da69157a72b1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0099f965b9fc4105ab98c565a68bc31c'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:30.315424 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-3f4a48b3-66d8-410d-b0af-da69157a72b1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0099f965b9fc4105ab98c565a68bc31c'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:30.339157 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-3f4a48b3-66d8-410d-b0af-da69157a72b1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0099f965b9fc4105ab98c565a68bc31c'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:30.359651 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-3f4a48b3-66d8-410d-b0af-da69157a72b1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0099f965b9fc4105ab98c565a68bc31c'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:30.361427 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-3f4a48b3-66d8-410d-b0af-da69157a72b1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0099f965b9fc4105ab98c565a68bc31c'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:30.389728 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 870/1728] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 00:55:30 2022] DELETE /identity/v3/users/0099f965b9fc4105ab98c565a68bc31c => generated 0 bytes in 249 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 28 00:55:30.396838 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d5bf836-1ec9-4864-b7c1-ec7d644d6509 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:30.397170 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d5bf836-1ec9-4864-b7c1-ec7d644d6509 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:30.397494 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d5bf836-1ec9-4864-b7c1-ec7d644d6509 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:30.420550 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-2d5bf836-1ec9-4864-b7c1-ec7d644d6509 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:55:30.422125 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-2d5bf836-1ec9-4864-b7c1-ec7d644d6509 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:30.477542 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:55:30.491533 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f4a48b3-66d8-410d-b0af-da69157a72b1 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:30.494414 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-3f4a48b3-66d8-410d-b0af-da69157a72b1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:30.568508 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 859/1729] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:55:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:55:30.581244 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d5bf836-1ec9-4864-b7c1-ec7d644d6509 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:30.583989 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-2d5bf836-1ec9-4864-b7c1-ec7d644d6509 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:30.620408 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f4a48b3-66d8-410d-b0af-da69157a72b1 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:30.625419 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cba15d0-91a3-4d36-a46c-9b6753e99c44 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:30.626120 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cba15d0-91a3-4d36-a46c-9b6753e99c44 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:30.626403 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cba15d0-91a3-4d36-a46c-9b6753e99c44 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:30.626682 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cba15d0-91a3-4d36-a46c-9b6753e99c44 admin admin] PATH_INFO: `/v3/projects/24a84ac76bb84719a5c6110fc4015f09` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:30.627869 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cba15d0-91a3-4d36-a46c-9b6753e99c44 admin admin] RBAC: Authorizing `identity:delete_project(project_id=24a84ac76bb84719a5c6110fc4015f09)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:30.630406 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cba15d0-91a3-4d36-a46c-9b6753e99c44 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:30.670332 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d5bf836-1ec9-4864-b7c1-ec7d644d6509 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:30.675456 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9466b7b3-41ee-429e-8c2a-a01f3c9ef2b8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:30.676191 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9466b7b3-41ee-429e-8c2a-a01f3c9ef2b8 service neutron] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:30.676511 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9466b7b3-41ee-429e-8c2a-a01f3c9ef2b8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:30.677035 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9466b7b3-41ee-429e-8c2a-a01f3c9ef2b8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:30.688644 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9466b7b3-41ee-429e-8c2a-a01f3c9ef2b8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:30.690327 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9466b7b3-41ee-429e-8c2a-a01f3c9ef2b8 service neutron] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:30.692709 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-2cba15d0-91a3-4d36-a46c-9b6753e99c44 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '24a84ac76bb84719a5c6110fc4015f09', 'request_id': 'req-2cba15d0-91a3-4d36-a46c-9b6753e99c44'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:30.698808 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-2cba15d0-91a3-4d36-a46c-9b6753e99c44 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '24a84ac76bb84719a5c6110fc4015f09', 'request_id': 'req-2cba15d0-91a3-4d36-a46c-9b6753e99c44'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:30.701211 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 860/1730] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:55:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 121 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:55:30.706473 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-2cba15d0-91a3-4d36-a46c-9b6753e99c44 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '24a84ac76bb84719a5c6110fc4015f09', 'request_id': 'req-2cba15d0-91a3-4d36-a46c-9b6753e99c44'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:30.717616 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:55:30.721988 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-2cba15d0-91a3-4d36-a46c-9b6753e99c44 admin admin] The token cache is being invalidate because project 24a84ac76bb84719a5c6110fc4015f09 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81508) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 00:55:30.722804 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-2cba15d0-91a3-4d36-a46c-9b6753e99c44 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:30.723963 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-2cba15d0-91a3-4d36-a46c-9b6753e99c44 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:30.726115 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 871/1731] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 00:55:30 2022] DELETE /identity/v3/projects/24a84ac76bb84719a5c6110fc4015f09 => generated 0 bytes in 236 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 28 00:55:30.730585 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9466b7b3-41ee-429e-8c2a-a01f3c9ef2b8 service neutron] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:30.741046 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9466b7b3-41ee-429e-8c2a-a01f3c9ef2b8 service neutron] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:30.805635 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70542225-1e39-451e-ad81-4392c7066ab1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:30.806288 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70542225-1e39-451e-ad81-4392c7066ab1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:30.806579 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70542225-1e39-451e-ad81-4392c7066ab1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:30.806839 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70542225-1e39-451e-ad81-4392c7066ab1 admin admin] PATH_INFO: `/v3/users/80139101c3154069a094e7cb56bcc565` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:30.808211 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70542225-1e39-451e-ad81-4392c7066ab1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=80139101c3154069a094e7cb56bcc565)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:30.810119 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70542225-1e39-451e-ad81-4392c7066ab1 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:30.864556 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-70542225-1e39-451e-ad81-4392c7066ab1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '80139101c3154069a094e7cb56bcc565'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:30.878871 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-70542225-1e39-451e-ad81-4392c7066ab1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '80139101c3154069a094e7cb56bcc565'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:30.887092 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-70542225-1e39-451e-ad81-4392c7066ab1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '80139101c3154069a094e7cb56bcc565'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:30.894579 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-70542225-1e39-451e-ad81-4392c7066ab1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '80139101c3154069a094e7cb56bcc565'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:30.895293 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-70542225-1e39-451e-ad81-4392c7066ab1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '80139101c3154069a094e7cb56bcc565'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:30.897474 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:55:30.903167 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 861/1732] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 00:55:30 2022] DELETE /identity/v3/users/80139101c3154069a094e7cb56bcc565 => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 28 00:55:30.993163 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cba15d0-91a3-4d36-a46c-9b6753e99c44 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:30.994536 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-2cba15d0-91a3-4d36-a46c-9b6753e99c44 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:31.010372 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cba15d0-91a3-4d36-a46c-9b6753e99c44 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:31.077626 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e864112-3b59-473c-bea8-b3204393ca69 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:31.078651 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e864112-3b59-473c-bea8-b3204393ca69 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:31.078924 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e864112-3b59-473c-bea8-b3204393ca69 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:31.079196 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e864112-3b59-473c-bea8-b3204393ca69 admin admin] PATH_INFO: `/v3/projects/1782b3b927be492a9255a0c25d773d34` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:31.080313 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e864112-3b59-473c-bea8-b3204393ca69 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1782b3b927be492a9255a0c25d773d34)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:31.082258 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e864112-3b59-473c-bea8-b3204393ca69 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:31.140190 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-4e864112-3b59-473c-bea8-b3204393ca69 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1782b3b927be492a9255a0c25d773d34', 'request_id': 'req-4e864112-3b59-473c-bea8-b3204393ca69'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:31.157628 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-4e864112-3b59-473c-bea8-b3204393ca69 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1782b3b927be492a9255a0c25d773d34', 'request_id': 'req-4e864112-3b59-473c-bea8-b3204393ca69'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:31.165553 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-4e864112-3b59-473c-bea8-b3204393ca69 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1782b3b927be492a9255a0c25d773d34', 'request_id': 'req-4e864112-3b59-473c-bea8-b3204393ca69'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:31.182405 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-4e864112-3b59-473c-bea8-b3204393ca69 admin admin] The token cache is being invalidate because project 1782b3b927be492a9255a0c25d773d34 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81508) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 00:55:31.182756 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-4e864112-3b59-473c-bea8-b3204393ca69 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:31.183392 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-4e864112-3b59-473c-bea8-b3204393ca69 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:31.185312 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 872/1733] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 00:55:30 2022] DELETE /identity/v3/projects/1782b3b927be492a9255a0c25d773d34 => generated 0 bytes in 193 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:55:31.189967 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d28308f-13a0-4364-9004-fac6359ec667 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:31.190449 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d28308f-13a0-4364-9004-fac6359ec667 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:31.190536 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d28308f-13a0-4364-9004-fac6359ec667 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:31.218718 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-1d28308f-13a0-4364-9004-fac6359ec667 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:55:31.220077 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-1d28308f-13a0-4364-9004-fac6359ec667 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:31.310015 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 862/1734] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:55:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:55:31.317996 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4353fed-c49f-495b-951b-181eeddf41dc None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:31.318283 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4353fed-c49f-495b-951b-181eeddf41dc None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:31.318561 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4353fed-c49f-495b-951b-181eeddf41dc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:31.333080 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d28308f-13a0-4364-9004-fac6359ec667 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:31.334885 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-1d28308f-13a0-4364-9004-fac6359ec667 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:31.343498 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-b4353fed-c49f-495b-951b-181eeddf41dc None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:55:31.344919 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-b4353fed-c49f-495b-951b-181eeddf41dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:31.348235 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d28308f-13a0-4364-9004-fac6359ec667 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:31.353338 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b942237-4de4-410e-889a-07cc52a22983 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:31.353852 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 873/1735] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:55:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:55:31.357491 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b942237-4de4-410e-889a-07cc52a22983 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:31.357845 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b942237-4de4-410e-889a-07cc52a22983 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:31.358187 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b942237-4de4-410e-889a-07cc52a22983 admin admin] PATH_INFO: `/v3/projects/ef293af3c6344f448140cf090632af81` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:31.358407 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4353fed-c49f-495b-951b-181eeddf41dc None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:31.359639 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b942237-4de4-410e-889a-07cc52a22983 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ef293af3c6344f448140cf090632af81)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:31.361762 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b942237-4de4-410e-889a-07cc52a22983 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:31.392992 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4353fed-c49f-495b-951b-181eeddf41dc None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:31.392992 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68679622-aa1f-4647-b104-ec46cbba13b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:31.392992 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68679622-aa1f-4647-b104-ec46cbba13b5 admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:31.392992 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68679622-aa1f-4647-b104-ec46cbba13b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:31.393386 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68679622-aa1f-4647-b104-ec46cbba13b5 admin admin] PATH_INFO: `/v3/domains` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:31.393386 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68679622-aa1f-4647-b104-ec46cbba13b5 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:55:31.393386 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68679622-aa1f-4647-b104-ec46cbba13b5 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:31.393386 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68679622-aa1f-4647-b104-ec46cbba13b5 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:31.413175 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 874/1736] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Aug 28 00:55:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 28 00:55:31.414287 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68679622-aa1f-4647-b104-ec46cbba13b5 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:31.427773 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-4b942237-4de4-410e-889a-07cc52a22983 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ef293af3c6344f448140cf090632af81', 'request_id': 'req-4b942237-4de4-410e-889a-07cc52a22983'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:31.432546 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68679622-aa1f-4647-b104-ec46cbba13b5 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:31.434549 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-4b942237-4de4-410e-889a-07cc52a22983 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ef293af3c6344f448140cf090632af81', 'request_id': 'req-4b942237-4de4-410e-889a-07cc52a22983'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:31.447451 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-4b942237-4de4-410e-889a-07cc52a22983 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ef293af3c6344f448140cf090632af81', 'request_id': 'req-4b942237-4de4-410e-889a-07cc52a22983'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:31.466078 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-4b942237-4de4-410e-889a-07cc52a22983 admin admin] The token cache is being invalidate because project ef293af3c6344f448140cf090632af81 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81507) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 00:55:31.466690 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-4b942237-4de4-410e-889a-07cc52a22983 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:31.467664 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-4b942237-4de4-410e-889a-07cc52a22983 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:31.472761 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 863/1737] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 00:55:31 2022] DELETE /identity/v3/projects/ef293af3c6344f448140cf090632af81 => generated 0 bytes in 140 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 28 00:55:31.481228 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-727faaaa-ce0c-4081-b635-3d0308103d18 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:31.482872 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-727faaaa-ce0c-4081-b635-3d0308103d18 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:31.483330 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-727faaaa-ce0c-4081-b635-3d0308103d18 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:31.518859 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-727faaaa-ce0c-4081-b635-3d0308103d18 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:55:31.520510 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-727faaaa-ce0c-4081-b635-3d0308103d18 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:31.548597 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8756e0d9-0100-4410-b95b-f0a924400370 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:31.549251 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8756e0d9-0100-4410-b95b-f0a924400370 admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:31.549532 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8756e0d9-0100-4410-b95b-f0a924400370 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:31.549803 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8756e0d9-0100-4410-b95b-f0a924400370 admin admin] PATH_INFO: `/v3/projects` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:31.550444 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8756e0d9-0100-4410-b95b-f0a924400370 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:31.552553 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8756e0d9-0100-4410-b95b-f0a924400370 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:31.576387 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:55:31.599714 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 875/1738] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 00:55:31 2022] POST /identity/v3/projects => generated 379 bytes in 186 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:55:31.600756 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8756e0d9-0100-4410-b95b-f0a924400370 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:31.603091 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-8756e0d9-0100-4410-b95b-f0a924400370 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:31.669077 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 864/1739] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:55:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 28 00:55:31.676085 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f6937f-fe02-48c5-a9ac-62b6d09bc915 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:31.677386 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f6937f-fe02-48c5-a9ac-62b6d09bc915 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:31.678001 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f6937f-fe02-48c5-a9ac-62b6d09bc915 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:31.722570 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8756e0d9-0100-4410-b95b-f0a924400370 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:31.730109 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-49f6937f-fe02-48c5-a9ac-62b6d09bc915 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:55:31.731704 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-49f6937f-fe02-48c5-a9ac-62b6d09bc915 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:31.743828 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e446290f-d8e2-434c-a3a1-94a1e5d21c71 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:31.744148 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e446290f-d8e2-434c-a3a1-94a1e5d21c71 admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:31.744509 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e446290f-d8e2-434c-a3a1-94a1e5d21c71 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:31.744857 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e446290f-d8e2-434c-a3a1-94a1e5d21c71 admin admin] PATH_INFO: `/v3/users` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:31.745582 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e446290f-d8e2-434c-a3a1-94a1e5d21c71 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:31.747559 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e446290f-d8e2-434c-a3a1-94a1e5d21c71 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:31.759113 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 865/1740] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:55:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:55:31.760877 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49f6937f-fe02-48c5-a9ac-62b6d09bc915 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:31.779696 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49f6937f-fe02-48c5-a9ac-62b6d09bc915 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:31.789651 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2a9f9d2-a021-49d3-bc87-a862a3855be4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:31.790389 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a9f9d2-a021-49d3-bc87-a862a3855be4 admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:31.790737 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a9f9d2-a021-49d3-bc87-a862a3855be4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:31.792770 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a9f9d2-a021-49d3-bc87-a862a3855be4 admin admin] PATH_INFO: `/v3/domains` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:31.795095 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2a9f9d2-a021-49d3-bc87-a862a3855be4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:55:31.795095 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2a9f9d2-a021-49d3-bc87-a862a3855be4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:31.797790 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2a9f9d2-a021-49d3-bc87-a862a3855be4 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:31.807806 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 876/1741] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 00:55:31 2022] POST /identity/v3/users => generated 290 bytes in 205 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:55:31.809180 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e446290f-d8e2-434c-a3a1-94a1e5d21c71 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:31.817433 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 866/1742] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Aug 28 00:55:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 57 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 28 00:55:31.822138 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2a9f9d2-a021-49d3-bc87-a862a3855be4 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:31.827073 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e446290f-d8e2-434c-a3a1-94a1e5d21c71 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:31.832324 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:55:31.837502 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa89619a-a656-40c0-9681-eb466ef61081 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:31.837875 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2a9f9d2-a021-49d3-bc87-a862a3855be4 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:31.838886 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa89619a-a656-40c0-9681-eb466ef61081 admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:31.841118 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa89619a-a656-40c0-9681-eb466ef61081 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:31.841118 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa89619a-a656-40c0-9681-eb466ef61081 admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:31.841118 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa89619a-a656-40c0-9681-eb466ef61081 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:31.843308 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa89619a-a656-40c0-9681-eb466ef61081 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:31.851571 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be2aee04-3640-4b3d-8799-d172bc61a245 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:31.852735 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be2aee04-3640-4b3d-8799-d172bc61a245 admin admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:31.853275 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be2aee04-3640-4b3d-8799-d172bc61a245 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:31.854032 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be2aee04-3640-4b3d-8799-d172bc61a245 admin admin] PATH_INFO: `/v3/projects` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:31.854608 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 877/1743] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:55:31 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 28 00:55:31.855477 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be2aee04-3640-4b3d-8799-d172bc61a245 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:31.858529 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be2aee04-3640-4b3d-8799-d172bc61a245 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:31.860596 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa89619a-a656-40c0-9681-eb466ef61081 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:31.874977 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa89619a-a656-40c0-9681-eb466ef61081 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:31.883009 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26e0fd18-4799-45ef-abc6-f617a5b3d672 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:31.883910 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e0fd18-4799-45ef-abc6-f617a5b3d672 admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:31.884260 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e0fd18-4799-45ef-abc6-f617a5b3d672 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:31.884613 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e0fd18-4799-45ef-abc6-f617a5b3d672 admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:31.885486 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26e0fd18-4799-45ef-abc6-f617a5b3d672 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:31.887367 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26e0fd18-4799-45ef-abc6-f617a5b3d672 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:31.898195 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 867/1744] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 00:55:31 2022] POST /identity/v3/projects => generated 385 bytes in 77 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:55:31.902117 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be2aee04-3640-4b3d-8799-d172bc61a245 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:31.903169 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 878/1745] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:55:31 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 00:55:31.907463 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26e0fd18-4799-45ef-abc6-f617a5b3d672 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:31.922192 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26e0fd18-4799-45ef-abc6-f617a5b3d672 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:31.986483 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be2aee04-3640-4b3d-8799-d172bc61a245 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:31.991835 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebc86a57-0e44-4f60-b6bd-aa79b6d8f5a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:31.992490 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebc86a57-0e44-4f60-b6bd-aa79b6d8f5a2 admin admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:31.992795 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebc86a57-0e44-4f60-b6bd-aa79b6d8f5a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:31.993085 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebc86a57-0e44-4f60-b6bd-aa79b6d8f5a2 admin admin] PATH_INFO: `/v3/users` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:31.993726 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebc86a57-0e44-4f60-b6bd-aa79b6d8f5a2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:31.996070 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebc86a57-0e44-4f60-b6bd-aa79b6d8f5a2 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:32.011095 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0638875f-4d71-4d9b-916b-03fa83908ad4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:32.012480 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0638875f-4d71-4d9b-916b-03fa83908ad4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:32.012480 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0638875f-4d71-4d9b-916b-03fa83908ad4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:32.012652 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0638875f-4d71-4d9b-916b-03fa83908ad4 admin admin] PATH_INFO: `/v3/projects/0f37945dd12b46b9afc46b199cfed080/users/83c38966ba804a59888b6be8bbb808ec/roles/0487ef7e943e4701915cb8ce0f2899d7` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:32.029172 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 868/1746] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 00:55:31 2022] POST /identity/v3/users => generated 293 bytes in 127 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:55:32.031565 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a973fd7-2a5f-4850-9d17-cd0aafcb2a17 None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:32.031820 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a973fd7-2a5f-4850-9d17-cd0aafcb2a17 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:32.032317 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a973fd7-2a5f-4850-9d17-cd0aafcb2a17 None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:32.032480 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0638875f-4d71-4d9b-916b-03fa83908ad4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0f37945dd12b46b9afc46b199cfed080, user_id=83c38966ba804a59888b6be8bbb808ec, role_id=0487ef7e943e4701915cb8ce0f2899d7)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:32.033963 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 869/1747] 10.222.0.42 () {54 vars in 921 bytes} [Sun Aug 28 00:55:32 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:55:32.034888 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a973fd7-2a5f-4850-9d17-cd0aafcb2a17 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:32.039747 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0638875f-4d71-4d9b-916b-03fa83908ad4 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:32.050445 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a973fd7-2a5f-4850-9d17-cd0aafcb2a17 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:32.057513 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dbdf134-1f71-4336-9450-4292e810ee86 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:32.058147 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dbdf134-1f71-4336-9450-4292e810ee86 admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:32.058441 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dbdf134-1f71-4336-9450-4292e810ee86 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:32.058714 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dbdf134-1f71-4336-9450-4292e810ee86 admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:32.061100 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dbdf134-1f71-4336-9450-4292e810ee86 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:32.062880 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dbdf134-1f71-4336-9450-4292e810ee86 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:32.078060 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 879/1748] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 00:55:31 2022] PUT /identity/v3/projects/0f37945dd12b46b9afc46b199cfed080/users/83c38966ba804a59888b6be8bbb808ec/roles/0487ef7e943e4701915cb8ce0f2899d7 => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 28 00:55:32.080219 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65f9dc29-ba49-4276-b711-be2659a12ec6 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:32.080545 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65f9dc29-ba49-4276-b711-be2659a12ec6 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:32.080885 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65f9dc29-ba49-4276-b711-be2659a12ec6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:32.083334 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 870/1749] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:55:32 2022] GET /identity/v3/roles => generated 1420 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 00:55:32.084326 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dbdf134-1f71-4336-9450-4292e810ee86 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:32.095436 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dbdf134-1f71-4336-9450-4292e810ee86 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:32.140398 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-65f9dc29-ba49-4276-b711-be2659a12ec6 None None] MFA Rules not processed for user `83c38966ba804a59888b6be8bbb808ec`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:55:32.143728 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-65f9dc29-ba49-4276-b711-be2659a12ec6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:32.211314 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:55:32.212954 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a42c206-9030-43d3-b550-ac5411f33b6d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:32.213676 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a42c206-9030-43d3-b550-ac5411f33b6d admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:32.215258 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a42c206-9030-43d3-b550-ac5411f33b6d admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:32.215558 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a42c206-9030-43d3-b550-ac5411f33b6d admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:32.218105 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a42c206-9030-43d3-b550-ac5411f33b6d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:32.219966 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a42c206-9030-43d3-b550-ac5411f33b6d admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:32.239359 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 871/1750] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:55:32 2022] GET /identity/v3/roles => generated 1420 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 00:55:32.242522 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39e4b61e-80b3-4531-8f0d-7379951298bf None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:32.242522 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39e4b61e-80b3-4531-8f0d-7379951298bf None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:32.242522 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39e4b61e-80b3-4531-8f0d-7379951298bf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:32.275985 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 880/1751] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:55:32 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 197 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 28 00:55:32.276937 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65f9dc29-ba49-4276-b711-be2659a12ec6 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:32.292377 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65f9dc29-ba49-4276-b711-be2659a12ec6 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:32.293985 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-39e4b61e-80b3-4531-8f0d-7379951298bf None None] MFA Rules not processed for user `168b07df50c24593ba73dbde5b037aed`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:55:32.295411 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-39e4b61e-80b3-4531-8f0d-7379951298bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:32.297558 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-659d5f74-6834-4296-bc69-e757bc06efa2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:32.298274 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-659d5f74-6834-4296-bc69-e757bc06efa2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:32.298622 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-659d5f74-6834-4296-bc69-e757bc06efa2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:32.298963 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-659d5f74-6834-4296-bc69-e757bc06efa2 admin admin] PATH_INFO: `/v3/projects/80c6a677ec0b4fa3acc2925dfa0826be/users/f12b1e0bd60048a38166b80fdce4aafd/roles/0487ef7e943e4701915cb8ce0f2899d7` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:32.329821 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-659d5f74-6834-4296-bc69-e757bc06efa2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=80c6a677ec0b4fa3acc2925dfa0826be, user_id=f12b1e0bd60048a38166b80fdce4aafd, role_id=0487ef7e943e4701915cb8ce0f2899d7)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:32.332605 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-659d5f74-6834-4296-bc69-e757bc06efa2 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:32.350149 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 881/1752] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 00:55:32 2022] PUT /identity/v3/projects/80c6a677ec0b4fa3acc2925dfa0826be/users/f12b1e0bd60048a38166b80fdce4aafd/roles/0487ef7e943e4701915cb8ce0f2899d7 => generated 0 bytes in 73 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:55:32.351304 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-659d5f74-6834-4296-bc69-e757bc06efa2 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:32.374966 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:55:32.406125 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 872/1753] 10.222.0.42 () {58 vars in 1018 bytes} [Sun Aug 28 00:55:32 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:55:32.407648 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16970f2d-c692-4059-a411-8efd582ae9a5 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:32.408001 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16970f2d-c692-4059-a411-8efd582ae9a5 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:32.408276 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16970f2d-c692-4059-a411-8efd582ae9a5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:32.442960 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-659d5f74-6834-4296-bc69-e757bc06efa2 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:32.444817 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-16970f2d-c692-4059-a411-8efd582ae9a5 None None] MFA Rules not processed for user `f12b1e0bd60048a38166b80fdce4aafd`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:55:32.446455 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-16970f2d-c692-4059-a411-8efd582ae9a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:32.448291 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-085b682f-d7f6-48d4-95f4-e55ce953602b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:32.449006 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-085b682f-d7f6-48d4-95f4-e55ce953602b admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:32.449854 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-085b682f-d7f6-48d4-95f4-e55ce953602b admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:32.450235 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-085b682f-d7f6-48d4-95f4-e55ce953602b admin admin] PATH_INFO: `/v3/projects` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:32.450928 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-085b682f-d7f6-48d4-95f4-e55ce953602b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:32.452846 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-085b682f-d7f6-48d4-95f4-e55ce953602b admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:32.484850 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 882/1754] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 00:55:32 2022] POST /identity/v3/projects => generated 381 bytes in 134 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:55:32.489671 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-085b682f-d7f6-48d4-95f4-e55ce953602b admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:32.534435 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 873/1755] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:55:32 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:55:32.535606 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16970f2d-c692-4059-a411-8efd582ae9a5 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:32.536898 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-16970f2d-c692-4059-a411-8efd582ae9a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:32.598117 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-085b682f-d7f6-48d4-95f4-e55ce953602b admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:32.603114 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be06201a-080f-49b3-99dd-53043847efff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:32.603747 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be06201a-080f-49b3-99dd-53043847efff admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:32.603928 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be06201a-080f-49b3-99dd-53043847efff admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:32.604174 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be06201a-080f-49b3-99dd-53043847efff admin admin] PATH_INFO: `/v3/users` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:32.604810 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be06201a-080f-49b3-99dd-53043847efff admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:32.606780 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be06201a-080f-49b3-99dd-53043847efff admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:32.622160 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16970f2d-c692-4059-a411-8efd582ae9a5 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:32.634604 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 883/1756] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 00:55:32 2022] POST /identity/v3/users => generated 297 bytes in 143 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:55:32.634604 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be06201a-080f-49b3-99dd-53043847efff admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:32.642315 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be06201a-080f-49b3-99dd-53043847efff admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:32.647550 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1572e63a-aea3-4e81-bb2f-f302a57fb151 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:32.648141 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1572e63a-aea3-4e81-bb2f-f302a57fb151 admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:32.648415 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1572e63a-aea3-4e81-bb2f-f302a57fb151 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:32.648682 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1572e63a-aea3-4e81-bb2f-f302a57fb151 admin admin] PATH_INFO: `/v3/projects` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:32.649336 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1572e63a-aea3-4e81-bb2f-f302a57fb151 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:32.651231 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1572e63a-aea3-4e81-bb2f-f302a57fb151 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:32.680100 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 884/1757] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 00:55:32 2022] POST /identity/v3/projects => generated 387 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:55:32.681039 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1572e63a-aea3-4e81-bb2f-f302a57fb151 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:32.686357 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:55:32.693420 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05c96303-b645-4d8e-818d-afca493036ab service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '168b07df50c24593ba73dbde5b037aed', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:32.693979 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05c96303-b645-4d8e-818d-afca493036ab service nova] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:32.694258 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05c96303-b645-4d8e-818d-afca493036ab service nova] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:32.694530 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05c96303-b645-4d8e-818d-afca493036ab service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:32.703961 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05c96303-b645-4d8e-818d-afca493036ab service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:32.705580 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05c96303-b645-4d8e-818d-afca493036ab service nova] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:32.716033 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 874/1758] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:55:32 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 181 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:55:32.717554 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05c96303-b645-4d8e-818d-afca493036ab service nova] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:32.723152 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:55:32.732547 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05c96303-b645-4d8e-818d-afca493036ab service nova] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:32.751690 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1572e63a-aea3-4e81-bb2f-f302a57fb151 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:32.757438 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87bb7984-744b-4a73-b6e0-d37275f222e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:32.758245 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87bb7984-744b-4a73-b6e0-d37275f222e8 admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:32.758554 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87bb7984-744b-4a73-b6e0-d37275f222e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:32.758826 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87bb7984-744b-4a73-b6e0-d37275f222e8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:32.759469 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87bb7984-744b-4a73-b6e0-d37275f222e8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:32.761336 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87bb7984-744b-4a73-b6e0-d37275f222e8 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:32.776499 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 885/1759] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:55:32 2022] GET /identity/v3/roles => generated 1420 bytes in 93 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 00:55:32.779014 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87bb7984-744b-4a73-b6e0-d37275f222e8 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:32.793839 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87bb7984-744b-4a73-b6e0-d37275f222e8 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:32.801504 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a65636a1-e5dd-4810-a71b-152af2fec339 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:32.802292 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a65636a1-e5dd-4810-a71b-152af2fec339 admin admin] REQUEST_METHOD: `PUT` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:32.802666 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a65636a1-e5dd-4810-a71b-152af2fec339 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:32.803043 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a65636a1-e5dd-4810-a71b-152af2fec339 admin admin] PATH_INFO: `/v3/projects/dc20b98585784a6f85512bc9ee86b498/users/3403a07dacb2470bb21e8cacabda8c10/roles/f08d6043261747e3b4763c744eef7dfb` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:32.819357 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7235406-1ad8-429d-9bc5-252d809e8477 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:32.820650 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7235406-1ad8-429d-9bc5-252d809e8477 admin admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:32.821213 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7235406-1ad8-429d-9bc5-252d809e8477 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:32.821737 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7235406-1ad8-429d-9bc5-252d809e8477 admin admin] PATH_INFO: `/v3/users` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:32.822688 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7235406-1ad8-429d-9bc5-252d809e8477 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:32.825449 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7235406-1ad8-429d-9bc5-252d809e8477 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:32.826109 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a65636a1-e5dd-4810-a71b-152af2fec339 admin admin] RBAC: Authorizing `identity:create_grant(project_id=dc20b98585784a6f85512bc9ee86b498, user_id=3403a07dacb2470bb21e8cacabda8c10, role_id=f08d6043261747e3b4763c744eef7dfb)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:32.829030 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a65636a1-e5dd-4810-a71b-152af2fec339 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:32.847405 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 886/1760] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 00:55:32 2022] PUT /identity/v3/projects/dc20b98585784a6f85512bc9ee86b498/users/3403a07dacb2470bb21e8cacabda8c10/roles/f08d6043261747e3b4763c744eef7dfb => generated 0 bytes in 70 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 28 00:55:32.851265 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a65636a1-e5dd-4810-a71b-152af2fec339 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:32.857535 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 875/1761] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 00:55:32 2022] POST /identity/v3/users => generated 300 bytes in 138 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:55:32.859595 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7235406-1ad8-429d-9bc5-252d809e8477 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:32.871189 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7235406-1ad8-429d-9bc5-252d809e8477 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:32.927235 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a65636a1-e5dd-4810-a71b-152af2fec339 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:32.932005 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62ac7149-167e-473e-8bea-1d63143891ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:32.932872 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62ac7149-167e-473e-8bea-1d63143891ed admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:32.933182 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62ac7149-167e-473e-8bea-1d63143891ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:32.933450 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62ac7149-167e-473e-8bea-1d63143891ed admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:32.934103 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cc45e8c-a3bf-423e-865b-5d854574526a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:32.934241 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62ac7149-167e-473e-8bea-1d63143891ed admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:32.934790 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc45e8c-a3bf-423e-865b-5d854574526a admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:32.935521 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc45e8c-a3bf-423e-865b-5d854574526a admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:32.935521 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc45e8c-a3bf-423e-865b-5d854574526a admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:32.935858 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62ac7149-167e-473e-8bea-1d63143891ed admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:32.936194 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cc45e8c-a3bf-423e-865b-5d854574526a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:32.937985 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cc45e8c-a3bf-423e-865b-5d854574526a admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:32.951471 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 887/1762] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:55:32 2022] GET /identity/v3/roles => generated 1420 bytes in 97 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 00:55:32.952207 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 876/1763] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:55:32 2022] GET /identity/v3/roles => generated 1420 bytes in 94 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 00:55:32.956206 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cc45e8c-a3bf-423e-865b-5d854574526a admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:32.956339 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62ac7149-167e-473e-8bea-1d63143891ed admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:32.976045 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62ac7149-167e-473e-8bea-1d63143891ed admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:32.976780 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cc45e8c-a3bf-423e-865b-5d854574526a admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:32.981713 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1ab4536-f209-4965-9e0a-4f2fca6a01b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:32.981849 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e354810-f6be-4743-90d3-c9bd08673a2f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:32.982386 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1ab4536-f209-4965-9e0a-4f2fca6a01b4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:32.982550 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e354810-f6be-4743-90d3-c9bd08673a2f admin admin] REQUEST_METHOD: `PUT` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:32.982680 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1ab4536-f209-4965-9e0a-4f2fca6a01b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:32.982783 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e354810-f6be-4743-90d3-c9bd08673a2f admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:32.982977 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1ab4536-f209-4965-9e0a-4f2fca6a01b4 admin admin] PATH_INFO: `/v3/projects/dc20b98585784a6f85512bc9ee86b498/users/3403a07dacb2470bb21e8cacabda8c10/roles/f08d6043261747e3b4763c744eef7dfb` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:32.983181 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e354810-f6be-4743-90d3-c9bd08673a2f admin admin] PATH_INFO: `/v3/projects/9b270fae1ca6449faa60a02a13a58ecc/users/2cb921840bab44fda27105c01696680f/roles/f08d6043261747e3b4763c744eef7dfb` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:32.996543 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1ab4536-f209-4965-9e0a-4f2fca6a01b4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=dc20b98585784a6f85512bc9ee86b498, user_id=3403a07dacb2470bb21e8cacabda8c10, role_id=f08d6043261747e3b4763c744eef7dfb)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:32.996543 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1ab4536-f209-4965-9e0a-4f2fca6a01b4 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:32.999179 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 888/1764] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 00:55:32 2022] PUT /identity/v3/projects/dc20b98585784a6f85512bc9ee86b498/users/3403a07dacb2470bb21e8cacabda8c10/roles/f08d6043261747e3b4763c744eef7dfb => generated 0 bytes in 43 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:55:32.999646 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e354810-f6be-4743-90d3-c9bd08673a2f admin admin] RBAC: Authorizing `identity:create_grant(project_id=9b270fae1ca6449faa60a02a13a58ecc, user_id=2cb921840bab44fda27105c01696680f, role_id=f08d6043261747e3b4763c744eef7dfb)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:33.002412 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e354810-f6be-4743-90d3-c9bd08673a2f admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:33.013414 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2f1ef3b-86bb-48b5-a3a2-441c49b1f73f None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:33.014231 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2f1ef3b-86bb-48b5-a3a2-441c49b1f73f None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:33.014231 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2f1ef3b-86bb-48b5-a3a2-441c49b1f73f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:33.019591 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 877/1765] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 00:55:32 2022] PUT /identity/v3/projects/9b270fae1ca6449faa60a02a13a58ecc/users/2cb921840bab44fda27105c01696680f/roles/f08d6043261747e3b4763c744eef7dfb => generated 0 bytes in 64 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:55:33.024542 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e354810-f6be-4743-90d3-c9bd08673a2f admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:33.045327 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-d2f1ef3b-86bb-48b5-a3a2-441c49b1f73f None None] MFA Rules not processed for user `3403a07dacb2470bb21e8cacabda8c10`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:55:33.046673 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-d2f1ef3b-86bb-48b5-a3a2-441c49b1f73f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:33.093439 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e354810-f6be-4743-90d3-c9bd08673a2f admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:33.095327 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:55:33.098554 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c2b88c9-8bcf-4a8a-927e-c0db8b27fd36 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:33.099136 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c2b88c9-8bcf-4a8a-927e-c0db8b27fd36 admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:33.099452 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c2b88c9-8bcf-4a8a-927e-c0db8b27fd36 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:33.099682 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c2b88c9-8bcf-4a8a-927e-c0db8b27fd36 admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:33.100435 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c2b88c9-8bcf-4a8a-927e-c0db8b27fd36 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:33.102252 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c2b88c9-8bcf-4a8a-927e-c0db8b27fd36 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:33.108989 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:55:33.120194 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 889/1766] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:55:33 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:55:33.130536 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 878/1767] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:55:33 2022] GET /identity/v3/roles => generated 1420 bytes in 103 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 28 00:55:33.131668 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2f1ef3b-86bb-48b5-a3a2-441c49b1f73f None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:33.133044 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c2b88c9-8bcf-4a8a-927e-c0db8b27fd36 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:33.135330 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-8c2b88c9-8bcf-4a8a-927e-c0db8b27fd36 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:33.143563 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2f1ef3b-86bb-48b5-a3a2-441c49b1f73f None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:33.148572 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-910dfef5-7c02-40b4-a79a-d02d4cb62610 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:33.149288 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-910dfef5-7c02-40b4-a79a-d02d4cb62610 admin admin] REQUEST_METHOD: `PUT` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:33.149565 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-910dfef5-7c02-40b4-a79a-d02d4cb62610 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:33.149841 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-910dfef5-7c02-40b4-a79a-d02d4cb62610 admin admin] PATH_INFO: `/v3/projects/9b270fae1ca6449faa60a02a13a58ecc/users/2cb921840bab44fda27105c01696680f/roles/f08d6043261747e3b4763c744eef7dfb` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:33.151803 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-910dfef5-7c02-40b4-a79a-d02d4cb62610 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9b270fae1ca6449faa60a02a13a58ecc, user_id=2cb921840bab44fda27105c01696680f, role_id=f08d6043261747e3b4763c744eef7dfb)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:33.154509 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-910dfef5-7c02-40b4-a79a-d02d4cb62610 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:33.165445 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 890/1768] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 00:55:33 2022] PUT /identity/v3/projects/9b270fae1ca6449faa60a02a13a58ecc/users/2cb921840bab44fda27105c01696680f/roles/f08d6043261747e3b4763c744eef7dfb => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:55:33.176313 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a41e512-1c65-4364-9289-401c485394e9 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:33.176594 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a41e512-1c65-4364-9289-401c485394e9 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:33.176871 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a41e512-1c65-4364-9289-401c485394e9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:33.206238 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-3a41e512-1c65-4364-9289-401c485394e9 None None] MFA Rules not processed for user `2cb921840bab44fda27105c01696680f`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:55:33.208134 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-3a41e512-1c65-4364-9289-401c485394e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:33.212101 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c2b88c9-8bcf-4a8a-927e-c0db8b27fd36 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:33.262159 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ffafc48-f78a-4e59-8fb6-94f9dfb10dd3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:33.263114 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ffafc48-f78a-4e59-8fb6-94f9dfb10dd3 service cinder] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:33.263114 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ffafc48-f78a-4e59-8fb6-94f9dfb10dd3 service cinder] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:33.263272 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ffafc48-f78a-4e59-8fb6-94f9dfb10dd3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:33.282876 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 891/1769] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:55:33 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 28 00:55:33.335952 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ffafc48-f78a-4e59-8fb6-94f9dfb10dd3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:33.337429 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ffafc48-f78a-4e59-8fb6-94f9dfb10dd3 service cinder] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:33.345561 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a41e512-1c65-4364-9289-401c485394e9 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:33.347217 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-3a41e512-1c65-4364-9289-401c485394e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:33.353180 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 879/1770] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:55:33 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 221 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 28 00:55:33.361562 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a41e512-1c65-4364-9289-401c485394e9 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:33.365794 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7f6af12-3e5e-4256-81e5-18b575f18e36 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:33.366407 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7f6af12-3e5e-4256-81e5-18b575f18e36 service cinder] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:33.366684 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7f6af12-3e5e-4256-81e5-18b575f18e36 service cinder] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:33.366956 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7f6af12-3e5e-4256-81e5-18b575f18e36 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:33.377007 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7f6af12-3e5e-4256-81e5-18b575f18e36 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:33.378432 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7f6af12-3e5e-4256-81e5-18b575f18e36 service cinder] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:33.388147 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 892/1771] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:55:33 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 28 00:55:33.944645 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ffafc48-f78a-4e59-8fb6-94f9dfb10dd3 service cinder] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:33.955797 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ffafc48-f78a-4e59-8fb6-94f9dfb10dd3 service cinder] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:33.962126 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fa96b1b-5aef-428b-952a-5ac59db8c397 tempest-VolumeQuotaClassesTest-1643371615 tempest-VolumeQuotaClassesTest-1643371615-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3403a07dacb2470bb21e8cacabda8c10', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc20b98585784a6f85512bc9ee86b498', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:33.963996 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fa96b1b-5aef-428b-952a-5ac59db8c397 tempest-VolumeQuotaClassesTest-1643371615 tempest-VolumeQuotaClassesTest-1643371615-project-admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:33.964477 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fa96b1b-5aef-428b-952a-5ac59db8c397 tempest-VolumeQuotaClassesTest-1643371615 tempest-VolumeQuotaClassesTest-1643371615-project-admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:33.964995 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fa96b1b-5aef-428b-952a-5ac59db8c397 tempest-VolumeQuotaClassesTest-1643371615 tempest-VolumeQuotaClassesTest-1643371615-project-admin] PATH_INFO: `/v3/domains` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:33.965936 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fa96b1b-5aef-428b-952a-5ac59db8c397 tempest-VolumeQuotaClassesTest-1643371615 tempest-VolumeQuotaClassesTest-1643371615-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:55:33.966454 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fa96b1b-5aef-428b-952a-5ac59db8c397 tempest-VolumeQuotaClassesTest-1643371615 tempest-VolumeQuotaClassesTest-1643371615-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:33.968608 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fa96b1b-5aef-428b-952a-5ac59db8c397 tempest-VolumeQuotaClassesTest-1643371615 tempest-VolumeQuotaClassesTest-1643371615-project-admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:33.997636 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 880/1772] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Aug 28 00:55:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:55:34.001891 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7f6af12-3e5e-4256-81e5-18b575f18e36 service cinder] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:34.012991 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7f6af12-3e5e-4256-81e5-18b575f18e36 service cinder] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:34.018394 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63a2a8d1-bb3e-45ac-bdb7-60e963f0784a tempest-VolumeQuotaClassesTest-1643371615 tempest-VolumeQuotaClassesTest-1643371615-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3403a07dacb2470bb21e8cacabda8c10', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc20b98585784a6f85512bc9ee86b498', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:34.019122 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63a2a8d1-bb3e-45ac-bdb7-60e963f0784a tempest-VolumeQuotaClassesTest-1643371615 tempest-VolumeQuotaClassesTest-1643371615-project-admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:34.019277 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63a2a8d1-bb3e-45ac-bdb7-60e963f0784a tempest-VolumeQuotaClassesTest-1643371615 tempest-VolumeQuotaClassesTest-1643371615-project-admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:34.019552 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63a2a8d1-bb3e-45ac-bdb7-60e963f0784a tempest-VolumeQuotaClassesTest-1643371615 tempest-VolumeQuotaClassesTest-1643371615-project-admin] PATH_INFO: `/v3/projects` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:34.020186 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63a2a8d1-bb3e-45ac-bdb7-60e963f0784a tempest-VolumeQuotaClassesTest-1643371615 tempest-VolumeQuotaClassesTest-1643371615-project-admin] RBAC: Authorizing `identity:create_project()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:34.022202 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63a2a8d1-bb3e-45ac-bdb7-60e963f0784a tempest-VolumeQuotaClassesTest-1643371615 tempest-VolumeQuotaClassesTest-1643371615-project-admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:34.053471 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 893/1773] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 00:55:34 2022] POST /identity/v3/projects => generated 354 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:55:34.058596 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fa96b1b-5aef-428b-952a-5ac59db8c397 tempest-VolumeQuotaClassesTest-1643371615 tempest-VolumeQuotaClassesTest-1643371615-project-admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:34.070874 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fa96b1b-5aef-428b-952a-5ac59db8c397 tempest-VolumeQuotaClassesTest-1643371615 tempest-VolumeQuotaClassesTest-1643371615-project-admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:34.132631 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:55:34.145624 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cddef595-25d4-4dfc-ad0f-aabddd55f04e tempest-VolumeQuotaClassesTest-1643371615 tempest-VolumeQuotaClassesTest-1643371615-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3403a07dacb2470bb21e8cacabda8c10', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc20b98585784a6f85512bc9ee86b498', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:34.146372 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cddef595-25d4-4dfc-ad0f-aabddd55f04e tempest-VolumeQuotaClassesTest-1643371615 tempest-VolumeQuotaClassesTest-1643371615-project-admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:34.146751 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cddef595-25d4-4dfc-ad0f-aabddd55f04e tempest-VolumeQuotaClassesTest-1643371615 tempest-VolumeQuotaClassesTest-1643371615-project-admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:34.147076 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cddef595-25d4-4dfc-ad0f-aabddd55f04e tempest-VolumeQuotaClassesTest-1643371615 tempest-VolumeQuotaClassesTest-1643371615-project-admin] PATH_INFO: `/v3/domains` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:34.147856 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cddef595-25d4-4dfc-ad0f-aabddd55f04e tempest-VolumeQuotaClassesTest-1643371615 tempest-VolumeQuotaClassesTest-1643371615-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:55:34.148790 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cddef595-25d4-4dfc-ad0f-aabddd55f04e tempest-VolumeQuotaClassesTest-1643371615 tempest-VolumeQuotaClassesTest-1643371615-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:34.149976 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cddef595-25d4-4dfc-ad0f-aabddd55f04e tempest-VolumeQuotaClassesTest-1643371615 tempest-VolumeQuotaClassesTest-1643371615-project-admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:34.187844 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 881/1774] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Aug 28 00:55:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 128 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:55:34.240283 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63a2a8d1-bb3e-45ac-bdb7-60e963f0784a tempest-VolumeQuotaClassesTest-1643371615 tempest-VolumeQuotaClassesTest-1643371615-project-admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:34.259007 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63a2a8d1-bb3e-45ac-bdb7-60e963f0784a tempest-VolumeQuotaClassesTest-1643371615 tempest-VolumeQuotaClassesTest-1643371615-project-admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:34.268937 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de39509f-1dcc-4c4a-83ec-8c60278a3a0b tempest-VolumeQuotaClassesTest-1643371615 tempest-VolumeQuotaClassesTest-1643371615-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3403a07dacb2470bb21e8cacabda8c10', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'dc20b98585784a6f85512bc9ee86b498', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:34.273559 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cddef595-25d4-4dfc-ad0f-aabddd55f04e tempest-VolumeQuotaClassesTest-1643371615 tempest-VolumeQuotaClassesTest-1643371615-project-admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:34.281558 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de39509f-1dcc-4c4a-83ec-8c60278a3a0b tempest-VolumeQuotaClassesTest-1643371615 tempest-VolumeQuotaClassesTest-1643371615-project-admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:34.281558 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de39509f-1dcc-4c4a-83ec-8c60278a3a0b tempest-VolumeQuotaClassesTest-1643371615 tempest-VolumeQuotaClassesTest-1643371615-project-admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:34.281558 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de39509f-1dcc-4c4a-83ec-8c60278a3a0b tempest-VolumeQuotaClassesTest-1643371615 tempest-VolumeQuotaClassesTest-1643371615-project-admin] PATH_INFO: `/v3/projects/62908c7f44f34de0a6e664131b3b80f2` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:34.282839 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de39509f-1dcc-4c4a-83ec-8c60278a3a0b tempest-VolumeQuotaClassesTest-1643371615 tempest-VolumeQuotaClassesTest-1643371615-project-admin] RBAC: Authorizing `identity:delete_project(project_id=62908c7f44f34de0a6e664131b3b80f2)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:34.287553 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de39509f-1dcc-4c4a-83ec-8c60278a3a0b tempest-VolumeQuotaClassesTest-1643371615 tempest-VolumeQuotaClassesTest-1643371615-project-admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:34.365390 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cddef595-25d4-4dfc-ad0f-aabddd55f04e tempest-VolumeQuotaClassesTest-1643371615 tempest-VolumeQuotaClassesTest-1643371615-project-admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:34.365643 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-de39509f-1dcc-4c4a-83ec-8c60278a3a0b tempest-VolumeQuotaClassesTest-1643371615 tempest-VolumeQuotaClassesTest-1643371615-project-admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '62908c7f44f34de0a6e664131b3b80f2', 'request_id': 'req-de39509f-1dcc-4c4a-83ec-8c60278a3a0b'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:34.372122 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-de39509f-1dcc-4c4a-83ec-8c60278a3a0b tempest-VolumeQuotaClassesTest-1643371615 tempest-VolumeQuotaClassesTest-1643371615-project-admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '62908c7f44f34de0a6e664131b3b80f2', 'request_id': 'req-de39509f-1dcc-4c4a-83ec-8c60278a3a0b'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:34.379926 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-de39509f-1dcc-4c4a-83ec-8c60278a3a0b tempest-VolumeQuotaClassesTest-1643371615 tempest-VolumeQuotaClassesTest-1643371615-project-admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '62908c7f44f34de0a6e664131b3b80f2', 'request_id': 'req-de39509f-1dcc-4c4a-83ec-8c60278a3a0b'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:34.398260 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-de39509f-1dcc-4c4a-83ec-8c60278a3a0b tempest-VolumeQuotaClassesTest-1643371615 tempest-VolumeQuotaClassesTest-1643371615-project-admin] The token cache is being invalidate because project 62908c7f44f34de0a6e664131b3b80f2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81508) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 00:55:34.398705 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-de39509f-1dcc-4c4a-83ec-8c60278a3a0b tempest-VolumeQuotaClassesTest-1643371615 tempest-VolumeQuotaClassesTest-1643371615-project-admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:34.399501 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-de39509f-1dcc-4c4a-83ec-8c60278a3a0b tempest-VolumeQuotaClassesTest-1643371615 tempest-VolumeQuotaClassesTest-1643371615-project-admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:34.401128 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 894/1775] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 00:55:34 2022] DELETE /identity/v3/projects/62908c7f44f34de0a6e664131b3b80f2 => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:55:34.465968 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cca461b-8d23-4492-ab43-3def64461e12 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:34.466778 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cca461b-8d23-4492-ab43-3def64461e12 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:34.467135 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cca461b-8d23-4492-ab43-3def64461e12 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:34.467407 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cca461b-8d23-4492-ab43-3def64461e12 admin admin] PATH_INFO: `/v3/users/f12b1e0bd60048a38166b80fdce4aafd` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:34.468557 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cca461b-8d23-4492-ab43-3def64461e12 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f12b1e0bd60048a38166b80fdce4aafd)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:34.471222 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cca461b-8d23-4492-ab43-3def64461e12 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:34.531519 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-8cca461b-8d23-4492-ab43-3def64461e12 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f12b1e0bd60048a38166b80fdce4aafd'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:34.544103 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-8cca461b-8d23-4492-ab43-3def64461e12 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f12b1e0bd60048a38166b80fdce4aafd'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:34.553426 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-8cca461b-8d23-4492-ab43-3def64461e12 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f12b1e0bd60048a38166b80fdce4aafd'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:34.561469 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-8cca461b-8d23-4492-ab43-3def64461e12 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f12b1e0bd60048a38166b80fdce4aafd'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:34.562186 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-8cca461b-8d23-4492-ab43-3def64461e12 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f12b1e0bd60048a38166b80fdce4aafd'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:34.568129 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 882/1776] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 00:55:34 2022] DELETE /identity/v3/users/f12b1e0bd60048a38166b80fdce4aafd => generated 0 bytes in 295 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:55:34.572589 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff2c9bd4-7459-4399-8e7d-29536f4666bd None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:34.572865 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff2c9bd4-7459-4399-8e7d-29536f4666bd None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:34.573157 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff2c9bd4-7459-4399-8e7d-29536f4666bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:34.581442 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=90,keystone:INSERT=3,keystone:DELETE=7 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:55:34.594912 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-ff2c9bd4-7459-4399-8e7d-29536f4666bd None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:55:34.597540 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-ff2c9bd4-7459-4399-8e7d-29536f4666bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:34.641418 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cca461b-8d23-4492-ab43-3def64461e12 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:34.643456 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-8cca461b-8d23-4492-ab43-3def64461e12 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:34.658946 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:55:34.665360 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 895/1777] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:55:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 28 00:55:34.675565 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff2c9bd4-7459-4399-8e7d-29536f4666bd None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:34.676937 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-ff2c9bd4-7459-4399-8e7d-29536f4666bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:34.712621 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cca461b-8d23-4492-ab43-3def64461e12 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:34.718084 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-318d7a2c-7393-4574-9994-6728141ce2c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:34.718823 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-318d7a2c-7393-4574-9994-6728141ce2c0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:34.719098 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-318d7a2c-7393-4574-9994-6728141ce2c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:34.719366 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-318d7a2c-7393-4574-9994-6728141ce2c0 admin admin] PATH_INFO: `/v3/users/83c38966ba804a59888b6be8bbb808ec` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:34.720523 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-318d7a2c-7393-4574-9994-6728141ce2c0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=83c38966ba804a59888b6be8bbb808ec)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:34.722551 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-318d7a2c-7393-4574-9994-6728141ce2c0 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:34.733280 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff2c9bd4-7459-4399-8e7d-29536f4666bd None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:34.737178 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7c80b07-4ffa-4c11-8a53-e874453744e7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:34.737799 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7c80b07-4ffa-4c11-8a53-e874453744e7 service neutron] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:34.738076 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7c80b07-4ffa-4c11-8a53-e874453744e7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:34.738351 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7c80b07-4ffa-4c11-8a53-e874453744e7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:34.748199 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7c80b07-4ffa-4c11-8a53-e874453744e7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:34.749659 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7c80b07-4ffa-4c11-8a53-e874453744e7 service neutron] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:34.759885 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 896/1778] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:55:34 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 28 00:55:34.771031 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-318d7a2c-7393-4574-9994-6728141ce2c0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '83c38966ba804a59888b6be8bbb808ec'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:34.782877 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-318d7a2c-7393-4574-9994-6728141ce2c0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '83c38966ba804a59888b6be8bbb808ec'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:34.791181 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-318d7a2c-7393-4574-9994-6728141ce2c0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '83c38966ba804a59888b6be8bbb808ec'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:34.798948 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-318d7a2c-7393-4574-9994-6728141ce2c0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '83c38966ba804a59888b6be8bbb808ec'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:34.799620 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-318d7a2c-7393-4574-9994-6728141ce2c0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '83c38966ba804a59888b6be8bbb808ec'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:34.805756 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 883/1779] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 00:55:34 2022] DELETE /identity/v3/users/83c38966ba804a59888b6be8bbb808ec => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 28 00:55:34.810474 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1aa32b9-202a-4169-bbba-fc46543bbcc7 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:34.810817 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1aa32b9-202a-4169-bbba-fc46543bbcc7 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:34.811092 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1aa32b9-202a-4169-bbba-fc46543bbcc7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:34.831285 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-e1aa32b9-202a-4169-bbba-fc46543bbcc7 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:55:34.832567 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-e1aa32b9-202a-4169-bbba-fc46543bbcc7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:34.896164 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 897/1780] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:55:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:55:34.906005 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-318d7a2c-7393-4574-9994-6728141ce2c0 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:34.907706 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-318d7a2c-7393-4574-9994-6728141ce2c0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:34.967490 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-318d7a2c-7393-4574-9994-6728141ce2c0 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:34.971944 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b24c896-8a37-4529-b89f-93389e2bb9d2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:34.972642 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b24c896-8a37-4529-b89f-93389e2bb9d2 service neutron] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:34.972869 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b24c896-8a37-4529-b89f-93389e2bb9d2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:34.973167 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b24c896-8a37-4529-b89f-93389e2bb9d2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:34.982735 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b24c896-8a37-4529-b89f-93389e2bb9d2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:34.984203 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b24c896-8a37-4529-b89f-93389e2bb9d2 service neutron] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:34.994054 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 884/1781] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:55:34 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:55:35.028554 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1aa32b9-202a-4169-bbba-fc46543bbcc7 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:35.030430 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-e1aa32b9-202a-4169-bbba-fc46543bbcc7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:35.044076 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1aa32b9-202a-4169-bbba-fc46543bbcc7 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:35.052719 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08c1256b-2801-4cd2-9c3b-a7750e5c6f9d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:35.058431 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08c1256b-2801-4cd2-9c3b-a7750e5c6f9d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:35.059327 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08c1256b-2801-4cd2-9c3b-a7750e5c6f9d admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:35.059950 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08c1256b-2801-4cd2-9c3b-a7750e5c6f9d admin admin] PATH_INFO: `/v3/projects/80c6a677ec0b4fa3acc2925dfa0826be` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:35.064252 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08c1256b-2801-4cd2-9c3b-a7750e5c6f9d admin admin] RBAC: Authorizing `identity:delete_project(project_id=80c6a677ec0b4fa3acc2925dfa0826be)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:35.066267 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08c1256b-2801-4cd2-9c3b-a7750e5c6f9d admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:35.113778 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:55:35.119142 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-08c1256b-2801-4cd2-9c3b-a7750e5c6f9d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '80c6a677ec0b4fa3acc2925dfa0826be', 'request_id': 'req-08c1256b-2801-4cd2-9c3b-a7750e5c6f9d'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:35.127617 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-08c1256b-2801-4cd2-9c3b-a7750e5c6f9d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '80c6a677ec0b4fa3acc2925dfa0826be', 'request_id': 'req-08c1256b-2801-4cd2-9c3b-a7750e5c6f9d'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:35.134376 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-08c1256b-2801-4cd2-9c3b-a7750e5c6f9d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '80c6a677ec0b4fa3acc2925dfa0826be', 'request_id': 'req-08c1256b-2801-4cd2-9c3b-a7750e5c6f9d'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:35.151462 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-08c1256b-2801-4cd2-9c3b-a7750e5c6f9d admin admin] The token cache is being invalidate because project 80c6a677ec0b4fa3acc2925dfa0826be was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81508) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 00:55:35.151749 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-08c1256b-2801-4cd2-9c3b-a7750e5c6f9d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:35.152412 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-08c1256b-2801-4cd2-9c3b-a7750e5c6f9d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:35.154358 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 898/1782] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 00:55:35 2022] DELETE /identity/v3/projects/80c6a677ec0b4fa3acc2925dfa0826be => generated 0 bytes in 126 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:55:35.158685 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b24c896-8a37-4529-b89f-93389e2bb9d2 service neutron] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:35.169036 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b24c896-8a37-4529-b89f-93389e2bb9d2 service neutron] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:35.206559 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:55:35.243460 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f52a35b-6781-4621-9af6-4da4991ac08d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:35.244327 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f52a35b-6781-4621-9af6-4da4991ac08d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:35.244643 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f52a35b-6781-4621-9af6-4da4991ac08d admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:35.244924 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f52a35b-6781-4621-9af6-4da4991ac08d admin admin] PATH_INFO: `/v3/users/2cb921840bab44fda27105c01696680f` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:35.246676 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f52a35b-6781-4621-9af6-4da4991ac08d admin admin] RBAC: Authorizing `identity:delete_user(user_id=2cb921840bab44fda27105c01696680f)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:35.248846 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f52a35b-6781-4621-9af6-4da4991ac08d admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:35.301910 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08c1256b-2801-4cd2-9c3b-a7750e5c6f9d admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:35.304595 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-08c1256b-2801-4cd2-9c3b-a7750e5c6f9d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:35.316308 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-0f52a35b-6781-4621-9af6-4da4991ac08d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2cb921840bab44fda27105c01696680f'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:35.325317 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08c1256b-2801-4cd2-9c3b-a7750e5c6f9d admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:35.332924 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d1e9028-97cd-4fcb-bb21-7659936d27e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:35.333880 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d1e9028-97cd-4fcb-bb21-7659936d27e3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:35.334157 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d1e9028-97cd-4fcb-bb21-7659936d27e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:35.334435 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d1e9028-97cd-4fcb-bb21-7659936d27e3 admin admin] PATH_INFO: `/v3/projects/0f37945dd12b46b9afc46b199cfed080` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:35.336559 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d1e9028-97cd-4fcb-bb21-7659936d27e3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0f37945dd12b46b9afc46b199cfed080)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:35.338751 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d1e9028-97cd-4fcb-bb21-7659936d27e3 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:35.342802 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-0f52a35b-6781-4621-9af6-4da4991ac08d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2cb921840bab44fda27105c01696680f'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:35.352093 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-0f52a35b-6781-4621-9af6-4da4991ac08d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2cb921840bab44fda27105c01696680f'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:35.362296 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-0f52a35b-6781-4621-9af6-4da4991ac08d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2cb921840bab44fda27105c01696680f'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:35.363021 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-0f52a35b-6781-4621-9af6-4da4991ac08d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2cb921840bab44fda27105c01696680f'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:35.371465 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 885/1783] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 00:55:35 2022] DELETE /identity/v3/users/2cb921840bab44fda27105c01696680f => generated 0 bytes in 212 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:55:35.400553 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-4d1e9028-97cd-4fcb-bb21-7659936d27e3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0f37945dd12b46b9afc46b199cfed080', 'request_id': 'req-4d1e9028-97cd-4fcb-bb21-7659936d27e3'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:35.408001 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-4d1e9028-97cd-4fcb-bb21-7659936d27e3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0f37945dd12b46b9afc46b199cfed080', 'request_id': 'req-4d1e9028-97cd-4fcb-bb21-7659936d27e3'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:35.416360 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-4d1e9028-97cd-4fcb-bb21-7659936d27e3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0f37945dd12b46b9afc46b199cfed080', 'request_id': 'req-4d1e9028-97cd-4fcb-bb21-7659936d27e3'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:35.432183 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-4d1e9028-97cd-4fcb-bb21-7659936d27e3 admin admin] The token cache is being invalidate because project 0f37945dd12b46b9afc46b199cfed080 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81508) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 00:55:35.432183 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-4d1e9028-97cd-4fcb-bb21-7659936d27e3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:35.432565 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-4d1e9028-97cd-4fcb-bb21-7659936d27e3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:35.434431 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 899/1784] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 00:55:35 2022] DELETE /identity/v3/projects/0f37945dd12b46b9afc46b199cfed080 => generated 0 bytes in 133 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:55:35.438741 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f52a35b-6781-4621-9af6-4da4991ac08d admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:35.440225 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-0f52a35b-6781-4621-9af6-4da4991ac08d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:35.550478 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f52a35b-6781-4621-9af6-4da4991ac08d admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:35.624773 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c1f19b8-dda1-43a3-969e-d12c3c53921c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:35.625594 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c1f19b8-dda1-43a3-969e-d12c3c53921c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:35.625947 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c1f19b8-dda1-43a3-969e-d12c3c53921c admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:35.626286 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c1f19b8-dda1-43a3-969e-d12c3c53921c admin admin] PATH_INFO: `/v3/users/3403a07dacb2470bb21e8cacabda8c10` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:35.627522 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c1f19b8-dda1-43a3-969e-d12c3c53921c admin admin] RBAC: Authorizing `identity:delete_user(user_id=3403a07dacb2470bb21e8cacabda8c10)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:35.629669 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c1f19b8-dda1-43a3-969e-d12c3c53921c admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:35.632482 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:55:35.689741 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-9c1f19b8-dda1-43a3-969e-d12c3c53921c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3403a07dacb2470bb21e8cacabda8c10'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:35.702352 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-9c1f19b8-dda1-43a3-969e-d12c3c53921c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3403a07dacb2470bb21e8cacabda8c10'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:35.710706 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-9c1f19b8-dda1-43a3-969e-d12c3c53921c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3403a07dacb2470bb21e8cacabda8c10'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:35.718030 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-9c1f19b8-dda1-43a3-969e-d12c3c53921c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3403a07dacb2470bb21e8cacabda8c10'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:35.718822 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-9c1f19b8-dda1-43a3-969e-d12c3c53921c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3403a07dacb2470bb21e8cacabda8c10'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:35.725405 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 886/1785] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 00:55:35 2022] DELETE /identity/v3/users/3403a07dacb2470bb21e8cacabda8c10 => generated 0 bytes in 287 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:55:35.864396 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d1e9028-97cd-4fcb-bb21-7659936d27e3 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:35.866681 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-4d1e9028-97cd-4fcb-bb21-7659936d27e3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:35.885656 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d1e9028-97cd-4fcb-bb21-7659936d27e3 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:35.965847 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32e69c1c-f33a-4822-b5b3-5462accc61fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:35.966487 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e69c1c-f33a-4822-b5b3-5462accc61fe admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:35.966765 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e69c1c-f33a-4822-b5b3-5462accc61fe admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:35.967035 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e69c1c-f33a-4822-b5b3-5462accc61fe admin admin] PATH_INFO: `/v3/projects/9b270fae1ca6449faa60a02a13a58ecc` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:35.968140 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32e69c1c-f33a-4822-b5b3-5462accc61fe admin admin] RBAC: Authorizing `identity:delete_project(project_id=9b270fae1ca6449faa60a02a13a58ecc)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:35.970223 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32e69c1c-f33a-4822-b5b3-5462accc61fe admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:36.010771 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c1f19b8-dda1-43a3-969e-d12c3c53921c admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:36.012236 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-9c1f19b8-dda1-43a3-969e-d12c3c53921c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:36.019225 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-32e69c1c-f33a-4822-b5b3-5462accc61fe admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9b270fae1ca6449faa60a02a13a58ecc', 'request_id': 'req-32e69c1c-f33a-4822-b5b3-5462accc61fe'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:36.026704 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c1f19b8-dda1-43a3-969e-d12c3c53921c admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:36.029154 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:55:36.031364 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-32e69c1c-f33a-4822-b5b3-5462accc61fe admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9b270fae1ca6449faa60a02a13a58ecc', 'request_id': 'req-32e69c1c-f33a-4822-b5b3-5462accc61fe'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:36.039602 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-32e69c1c-f33a-4822-b5b3-5462accc61fe admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9b270fae1ca6449faa60a02a13a58ecc', 'request_id': 'req-32e69c1c-f33a-4822-b5b3-5462accc61fe'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:36.053283 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-32e69c1c-f33a-4822-b5b3-5462accc61fe admin admin] The token cache is being invalidate because project 9b270fae1ca6449faa60a02a13a58ecc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81508) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 00:55:36.053565 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-32e69c1c-f33a-4822-b5b3-5462accc61fe admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:36.054173 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-32e69c1c-f33a-4822-b5b3-5462accc61fe admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:36.055863 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 900/1786] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 00:55:35 2022] DELETE /identity/v3/projects/9b270fae1ca6449faa60a02a13a58ecc => generated 0 bytes in 192 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:55:36.060667 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-547b0649-e2d9-4698-ac65-cb073e2a1a8e None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:36.060950 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-547b0649-e2d9-4698-ac65-cb073e2a1a8e None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:36.061250 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-547b0649-e2d9-4698-ac65-cb073e2a1a8e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:36.080640 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-547b0649-e2d9-4698-ac65-cb073e2a1a8e None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:55:36.081873 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-547b0649-e2d9-4698-ac65-cb073e2a1a8e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:36.091106 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-952770ca-9e1d-46b5-b486-8817411c06ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:36.091743 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-952770ca-9e1d-46b5-b486-8817411c06ff admin admin] REQUEST_METHOD: `DELETE` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:36.092029 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-952770ca-9e1d-46b5-b486-8817411c06ff admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:36.092306 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-952770ca-9e1d-46b5-b486-8817411c06ff admin admin] PATH_INFO: `/v3/projects/dc20b98585784a6f85512bc9ee86b498` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:36.093454 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-952770ca-9e1d-46b5-b486-8817411c06ff admin admin] RBAC: Authorizing `identity:delete_project(project_id=dc20b98585784a6f85512bc9ee86b498)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:36.095445 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-952770ca-9e1d-46b5-b486-8817411c06ff admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:36.145050 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-952770ca-9e1d-46b5-b486-8817411c06ff admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'dc20b98585784a6f85512bc9ee86b498', 'request_id': 'req-952770ca-9e1d-46b5-b486-8817411c06ff'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:36.148018 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 901/1787] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:55:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 28 00:55:36.150637 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-952770ca-9e1d-46b5-b486-8817411c06ff admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'dc20b98585784a6f85512bc9ee86b498', 'request_id': 'req-952770ca-9e1d-46b5-b486-8817411c06ff'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:36.157332 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-952770ca-9e1d-46b5-b486-8817411c06ff admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'dc20b98585784a6f85512bc9ee86b498', 'request_id': 'req-952770ca-9e1d-46b5-b486-8817411c06ff'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:36.157510 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99090cf1-e2a0-4bc1-9d0c-aa5820082746 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:36.157706 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99090cf1-e2a0-4bc1-9d0c-aa5820082746 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:36.157922 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99090cf1-e2a0-4bc1-9d0c-aa5820082746 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:36.171031 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-952770ca-9e1d-46b5-b486-8817411c06ff admin admin] The token cache is being invalidate because project dc20b98585784a6f85512bc9ee86b498 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81507) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 00:55:36.171333 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-952770ca-9e1d-46b5-b486-8817411c06ff admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:36.172113 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-952770ca-9e1d-46b5-b486-8817411c06ff admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:36.173864 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 887/1788] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 00:55:36 2022] DELETE /identity/v3/projects/dc20b98585784a6f85512bc9ee86b498 => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 28 00:55:36.178387 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6fc8250-a649-4dea-b37b-bbb3c6cb902f None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:36.178677 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6fc8250-a649-4dea-b37b-bbb3c6cb902f None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:36.178952 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6fc8250-a649-4dea-b37b-bbb3c6cb902f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:36.179227 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-99090cf1-e2a0-4bc1-9d0c-aa5820082746 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:55:36.180433 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-99090cf1-e2a0-4bc1-9d0c-aa5820082746 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:36.207482 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-d6fc8250-a649-4dea-b37b-bbb3c6cb902f None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:55:36.208753 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-d6fc8250-a649-4dea-b37b-bbb3c6cb902f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:36.215391 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:55:36.265342 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 902/1789] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:55:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 28 00:55:36.269667 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99090cf1-e2a0-4bc1-9d0c-aa5820082746 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:36.279012 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99090cf1-e2a0-4bc1-9d0c-aa5820082746 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:36.284679 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2af6487-e9a5-4b7a-980b-a443d1857d54 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:36.285426 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2af6487-e9a5-4b7a-980b-a443d1857d54 admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:36.285705 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2af6487-e9a5-4b7a-980b-a443d1857d54 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:36.286011 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2af6487-e9a5-4b7a-980b-a443d1857d54 admin admin] PATH_INFO: `/v3/domains` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:36.286694 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2af6487-e9a5-4b7a-980b-a443d1857d54 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:55:36.286992 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2af6487-e9a5-4b7a-980b-a443d1857d54 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:36.288681 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2af6487-e9a5-4b7a-980b-a443d1857d54 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:36.292372 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 888/1790] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:55:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 28 00:55:36.299240 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f35c325c-d995-4f3e-89c9-818d0308c421 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:36.299801 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f35c325c-d995-4f3e-89c9-818d0308c421 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:36.299801 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f35c325c-d995-4f3e-89c9-818d0308c421 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:36.303344 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 903/1791] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Aug 28 00:55:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 28 00:55:36.307416 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2af6487-e9a5-4b7a-980b-a443d1857d54 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:36.317964 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2af6487-e9a5-4b7a-980b-a443d1857d54 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:36.319944 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-f35c325c-d995-4f3e-89c9-818d0308c421 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:55:36.321201 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-f35c325c-d995-4f3e-89c9-818d0308c421 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:36.322683 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc36e8ee-ec8f-4568-b1c8-292f5a1dcda0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:36.323282 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc36e8ee-ec8f-4568-b1c8-292f5a1dcda0 admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:36.323560 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc36e8ee-ec8f-4568-b1c8-292f5a1dcda0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:36.323827 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc36e8ee-ec8f-4568-b1c8-292f5a1dcda0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:36.324775 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc36e8ee-ec8f-4568-b1c8-292f5a1dcda0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:36.326774 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc36e8ee-ec8f-4568-b1c8-292f5a1dcda0 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:36.330288 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 889/1792] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:55:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:55:36.334232 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f35c325c-d995-4f3e-89c9-818d0308c421 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:36.343260 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f35c325c-d995-4f3e-89c9-818d0308c421 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:36.348647 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8baea78-142d-48be-bca6-ec5a5d027d34 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:36.349345 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8baea78-142d-48be-bca6-ec5a5d027d34 admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:36.349623 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8baea78-142d-48be-bca6-ec5a5d027d34 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:36.349894 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8baea78-142d-48be-bca6-ec5a5d027d34 admin admin] PATH_INFO: `/v3/domains` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:36.350743 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8baea78-142d-48be-bca6-ec5a5d027d34 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:55:36.351044 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8baea78-142d-48be-bca6-ec5a5d027d34 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:36.352700 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8baea78-142d-48be-bca6-ec5a5d027d34 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:36.355290 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 904/1793] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 00:55:36 2022] POST /identity/v3/projects => generated 383 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 28 00:55:36.360340 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc36e8ee-ec8f-4568-b1c8-292f5a1dcda0 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:36.365229 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 890/1794] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Aug 28 00:55:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:55:36.370093 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8baea78-142d-48be-bca6-ec5a5d027d34 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:36.381760 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8baea78-142d-48be-bca6-ec5a5d027d34 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:36.414832 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=97,keystone:INSERT=2,keystone:DELETE=1 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:55:36.426612 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc36e8ee-ec8f-4568-b1c8-292f5a1dcda0 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:36.432772 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dac23f1b-37bb-4132-b3e7-9e83e644c2fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:36.433519 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dac23f1b-37bb-4132-b3e7-9e83e644c2fb admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:36.433834 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dac23f1b-37bb-4132-b3e7-9e83e644c2fb admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:36.434111 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dac23f1b-37bb-4132-b3e7-9e83e644c2fb admin admin] PATH_INFO: `/v3/users` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:36.434773 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dac23f1b-37bb-4132-b3e7-9e83e644c2fb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:36.436739 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dac23f1b-37bb-4132-b3e7-9e83e644c2fb admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:36.439774 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ab8af42-3856-4de5-a906-fb0d4e2a04e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:36.440392 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ab8af42-3856-4de5-a906-fb0d4e2a04e7 admin admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:36.440693 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ab8af42-3856-4de5-a906-fb0d4e2a04e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:36.440984 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ab8af42-3856-4de5-a906-fb0d4e2a04e7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:36.441634 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ab8af42-3856-4de5-a906-fb0d4e2a04e7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:36.443550 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ab8af42-3856-4de5-a906-fb0d4e2a04e7 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:36.462006 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 905/1795] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 00:55:36 2022] POST /identity/v3/users => generated 292 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 28 00:55:36.466980 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dac23f1b-37bb-4132-b3e7-9e83e644c2fb admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:36.479126 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dac23f1b-37bb-4132-b3e7-9e83e644c2fb admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:36.481388 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 891/1796] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 00:55:36 2022] POST /identity/v3/projects => generated 389 bytes in 112 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 28 00:55:36.486259 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ab8af42-3856-4de5-a906-fb0d4e2a04e7 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:36.555154 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c68485f-39af-4d3e-8273-09fc81e008cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:36.555794 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c68485f-39af-4d3e-8273-09fc81e008cc admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:36.556084 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c68485f-39af-4d3e-8273-09fc81e008cc admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:36.556501 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c68485f-39af-4d3e-8273-09fc81e008cc admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:36.557539 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c68485f-39af-4d3e-8273-09fc81e008cc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:36.559582 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c68485f-39af-4d3e-8273-09fc81e008cc admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:36.570985 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 906/1797] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:55:36 2022] GET /identity/v3/roles => generated 1420 bytes in 104 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 00:55:36.574931 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c68485f-39af-4d3e-8273-09fc81e008cc admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:36.583078 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ab8af42-3856-4de5-a906-fb0d4e2a04e7 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:36.587867 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f96597c8-dfa6-4d1e-b53c-a4bbb91c259a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:36.587996 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c68485f-39af-4d3e-8273-09fc81e008cc admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:36.588483 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f96597c8-dfa6-4d1e-b53c-a4bbb91c259a admin admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:36.588768 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f96597c8-dfa6-4d1e-b53c-a4bbb91c259a admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:36.589072 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f96597c8-dfa6-4d1e-b53c-a4bbb91c259a admin admin] PATH_INFO: `/v3/users` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:36.589710 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f96597c8-dfa6-4d1e-b53c-a4bbb91c259a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:36.591714 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f96597c8-dfa6-4d1e-b53c-a4bbb91c259a admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:36.595554 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96fca8c0-159a-469d-8aaa-461667af57f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:36.597229 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96fca8c0-159a-469d-8aaa-461667af57f6 admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:36.597682 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96fca8c0-159a-469d-8aaa-461667af57f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:36.597958 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96fca8c0-159a-469d-8aaa-461667af57f6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:36.599125 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96fca8c0-159a-469d-8aaa-461667af57f6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:36.602791 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96fca8c0-159a-469d-8aaa-461667af57f6 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:36.616148 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 907/1798] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:55:36 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 00:55:36.616925 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 892/1799] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 00:55:36 2022] POST /identity/v3/users => generated 295 bytes in 131 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 28 00:55:36.620653 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96fca8c0-159a-469d-8aaa-461667af57f6 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:36.622133 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f96597c8-dfa6-4d1e-b53c-a4bbb91c259a admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:36.631427 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96fca8c0-159a-469d-8aaa-461667af57f6 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:36.634270 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f96597c8-dfa6-4d1e-b53c-a4bbb91c259a admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:36.636158 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e495987-4e3e-4d30-a155-a537820a783a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:36.636893 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e495987-4e3e-4d30-a155-a537820a783a admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:36.637275 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e495987-4e3e-4d30-a155-a537820a783a admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:36.637770 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e495987-4e3e-4d30-a155-a537820a783a admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:36.638309 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e495987-4e3e-4d30-a155-a537820a783a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:36.639262 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5626e776-79b9-4c69-a53b-9d8bb1735a04 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:36.639927 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5626e776-79b9-4c69-a53b-9d8bb1735a04 admin admin] REQUEST_METHOD: `PUT` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:36.640277 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5626e776-79b9-4c69-a53b-9d8bb1735a04 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:36.640893 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5626e776-79b9-4c69-a53b-9d8bb1735a04 admin admin] PATH_INFO: `/v3/projects/a91de8ea5f65402d975e559d676388d9/users/434692f2bda14675b9d53756453cb7b9/roles/0487ef7e943e4701915cb8ce0f2899d7` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:36.643013 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:55:36.645872 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e495987-4e3e-4d30-a155-a537820a783a admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:36.659456 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5626e776-79b9-4c69-a53b-9d8bb1735a04 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a91de8ea5f65402d975e559d676388d9, user_id=434692f2bda14675b9d53756453cb7b9, role_id=0487ef7e943e4701915cb8ce0f2899d7)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:36.661979 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 908/1800] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:55:36 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 00:55:36.662890 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5626e776-79b9-4c69-a53b-9d8bb1735a04 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:36.663001 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e495987-4e3e-4d30-a155-a537820a783a admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:36.673747 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e495987-4e3e-4d30-a155-a537820a783a admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:36.680603 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 893/1801] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 00:55:36 2022] PUT /identity/v3/projects/a91de8ea5f65402d975e559d676388d9/users/434692f2bda14675b9d53756453cb7b9/roles/0487ef7e943e4701915cb8ce0f2899d7 => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:55:36.691801 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-451846b0-d3c7-4e04-ab23-600b9faa92ed None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:36.692099 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-451846b0-d3c7-4e04-ab23-600b9faa92ed None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:36.692380 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-451846b0-d3c7-4e04-ab23-600b9faa92ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:36.739679 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-451846b0-d3c7-4e04-ab23-600b9faa92ed None None] MFA Rules not processed for user `434692f2bda14675b9d53756453cb7b9`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:55:36.741135 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-451846b0-d3c7-4e04-ab23-600b9faa92ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:36.768804 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d7cb80f-e356-4efe-b25e-e1cb913c3bb6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:36.769463 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d7cb80f-e356-4efe-b25e-e1cb913c3bb6 admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:36.769747 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d7cb80f-e356-4efe-b25e-e1cb913c3bb6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:36.770028 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d7cb80f-e356-4efe-b25e-e1cb913c3bb6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:36.770676 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d7cb80f-e356-4efe-b25e-e1cb913c3bb6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:36.772400 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d7cb80f-e356-4efe-b25e-e1cb913c3bb6 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:36.783595 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 909/1802] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:55:36 2022] GET /identity/v3/roles => generated 1420 bytes in 121 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 00:55:36.787679 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d7cb80f-e356-4efe-b25e-e1cb913c3bb6 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:36.807282 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d7cb80f-e356-4efe-b25e-e1cb913c3bb6 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:36.813714 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e845b8a1-a103-4f9d-922c-ee3e16e31663 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:36.814530 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e845b8a1-a103-4f9d-922c-ee3e16e31663 admin admin] REQUEST_METHOD: `PUT` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:36.814943 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e845b8a1-a103-4f9d-922c-ee3e16e31663 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:36.815272 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e845b8a1-a103-4f9d-922c-ee3e16e31663 admin admin] PATH_INFO: `/v3/projects/0ec600f2e5e44dcf840e990f26c7d49f/users/432e8a2698dd478d9a04c2e0ff2ac458/roles/0487ef7e943e4701915cb8ce0f2899d7` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:36.833427 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e845b8a1-a103-4f9d-922c-ee3e16e31663 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0ec600f2e5e44dcf840e990f26c7d49f, user_id=432e8a2698dd478d9a04c2e0ff2ac458, role_id=0487ef7e943e4701915cb8ce0f2899d7)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:36.836142 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e845b8a1-a103-4f9d-922c-ee3e16e31663 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:36.838077 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 894/1803] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:55:36 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 147 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:55:36.843031 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-451846b0-d3c7-4e04-ab23-600b9faa92ed None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:36.854080 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 910/1804] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 00:55:36 2022] PUT /identity/v3/projects/0ec600f2e5e44dcf840e990f26c7d49f/users/432e8a2698dd478d9a04c2e0ff2ac458/roles/0487ef7e943e4701915cb8ce0f2899d7 => generated 0 bytes in 67 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 28 00:55:36.864179 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-451846b0-d3c7-4e04-ab23-600b9faa92ed None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:36.865462 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d9a8f01-eab0-43a5-bd09-812839b62aee None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:36.866397 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d9a8f01-eab0-43a5-bd09-812839b62aee None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:36.866397 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d9a8f01-eab0-43a5-bd09-812839b62aee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:36.904771 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-0d9a8f01-eab0-43a5-bd09-812839b62aee None None] MFA Rules not processed for user `432e8a2698dd478d9a04c2e0ff2ac458`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:55:36.906661 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-0d9a8f01-eab0-43a5-bd09-812839b62aee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:36.928899 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:55:36.957400 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc5bdbba-f896-40dd-983c-d1481cfa755c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:36.958216 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc5bdbba-f896-40dd-983c-d1481cfa755c admin admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:36.958502 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc5bdbba-f896-40dd-983c-d1481cfa755c admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:36.958770 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc5bdbba-f896-40dd-983c-d1481cfa755c admin admin] PATH_INFO: `/v3/projects` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:36.959457 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc5bdbba-f896-40dd-983c-d1481cfa755c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:36.961478 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc5bdbba-f896-40dd-983c-d1481cfa755c admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:37.013080 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 895/1805] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 00:55:36 2022] POST /identity/v3/projects => generated 383 bytes in 170 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 28 00:55:37.017400 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc5bdbba-f896-40dd-983c-d1481cfa755c admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:37.020299 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 911/1806] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:55:36 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 28 00:55:37.026181 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d9a8f01-eab0-43a5-bd09-812839b62aee None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:37.039765 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d9a8f01-eab0-43a5-bd09-812839b62aee None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:37.103099 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:55:37.107624 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc5bdbba-f896-40dd-983c-d1481cfa755c admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:37.113019 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5117ff5c-ae66-4855-93da-cce323d85c1c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:37.113922 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5117ff5c-ae66-4855-93da-cce323d85c1c admin admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:37.114321 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5117ff5c-ae66-4855-93da-cce323d85c1c admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:37.114681 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5117ff5c-ae66-4855-93da-cce323d85c1c admin admin] PATH_INFO: `/v3/users` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:37.115679 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5117ff5c-ae66-4855-93da-cce323d85c1c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:37.117510 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5117ff5c-ae66-4855-93da-cce323d85c1c admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:37.119960 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32d906b9-ddc3-4e00-93af-99ecc6aef67a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:37.120720 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32d906b9-ddc3-4e00-93af-99ecc6aef67a admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:37.121037 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32d906b9-ddc3-4e00-93af-99ecc6aef67a admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:37.121311 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32d906b9-ddc3-4e00-93af-99ecc6aef67a admin admin] PATH_INFO: `/v3/projects` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:37.121986 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32d906b9-ddc3-4e00-93af-99ecc6aef67a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:37.124095 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32d906b9-ddc3-4e00-93af-99ecc6aef67a admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:37.142647 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 896/1807] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 00:55:37 2022] POST /identity/v3/users => generated 298 bytes in 126 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:55:37.146686 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5117ff5c-ae66-4855-93da-cce323d85c1c admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:37.157920 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 912/1808] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 00:55:37 2022] POST /identity/v3/projects => generated 387 bytes in 130 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 28 00:55:37.158100 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5117ff5c-ae66-4855-93da-cce323d85c1c admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:37.158926 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32d906b9-ddc3-4e00-93af-99ecc6aef67a admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:37.216363 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-660f1f97-3379-44b5-9bb3-5ab27e39badb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:37.217005 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-660f1f97-3379-44b5-9bb3-5ab27e39badb admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:37.217288 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-660f1f97-3379-44b5-9bb3-5ab27e39badb admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:37.217555 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-660f1f97-3379-44b5-9bb3-5ab27e39badb admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:37.218204 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-660f1f97-3379-44b5-9bb3-5ab27e39badb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:37.219834 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-660f1f97-3379-44b5-9bb3-5ab27e39badb admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:37.222817 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32d906b9-ddc3-4e00-93af-99ecc6aef67a admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:37.227307 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dcf90d4-dc8e-4b4f-9c95-758b96d2d928 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:37.227910 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dcf90d4-dc8e-4b4f-9c95-758b96d2d928 admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:37.228183 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dcf90d4-dc8e-4b4f-9c95-758b96d2d928 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:37.228452 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dcf90d4-dc8e-4b4f-9c95-758b96d2d928 admin admin] PATH_INFO: `/v3/users` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:37.229124 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dcf90d4-dc8e-4b4f-9c95-758b96d2d928 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:37.232889 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 897/1809] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:55:37 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 28 00:55:37.233064 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dcf90d4-dc8e-4b4f-9c95-758b96d2d928 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:37.233973 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-660f1f97-3379-44b5-9bb3-5ab27e39badb admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:37.244491 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-660f1f97-3379-44b5-9bb3-5ab27e39badb admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:37.249759 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-327ad51c-4d36-48db-a6e1-6a9202983e1c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:37.250433 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-327ad51c-4d36-48db-a6e1-6a9202983e1c admin admin] REQUEST_METHOD: `PUT` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:37.250724 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-327ad51c-4d36-48db-a6e1-6a9202983e1c admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:37.250972 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-327ad51c-4d36-48db-a6e1-6a9202983e1c admin admin] PATH_INFO: `/v3/projects/0f0baac91140411c9f2f1509031d3ff3/users/1ff9bd07b1f649ea8535d6cd8750bfec/roles/f08d6043261747e3b4763c744eef7dfb` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:37.254419 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 913/1810] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 00:55:37 2022] POST /identity/v3/users => generated 300 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 28 00:55:37.258422 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dcf90d4-dc8e-4b4f-9c95-758b96d2d928 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:37.268340 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dcf90d4-dc8e-4b4f-9c95-758b96d2d928 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:37.268983 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-327ad51c-4d36-48db-a6e1-6a9202983e1c admin admin] RBAC: Authorizing `identity:create_grant(project_id=0f0baac91140411c9f2f1509031d3ff3, user_id=1ff9bd07b1f649ea8535d6cd8750bfec, role_id=f08d6043261747e3b4763c744eef7dfb)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:37.271650 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-327ad51c-4d36-48db-a6e1-6a9202983e1c admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:37.273416 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1413111c-bba2-4fc3-89d4-8b6145626892 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:37.274307 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1413111c-bba2-4fc3-89d4-8b6145626892 admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:37.274565 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1413111c-bba2-4fc3-89d4-8b6145626892 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:37.274837 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1413111c-bba2-4fc3-89d4-8b6145626892 admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:37.275476 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1413111c-bba2-4fc3-89d4-8b6145626892 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:37.277383 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1413111c-bba2-4fc3-89d4-8b6145626892 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:37.286249 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 898/1811] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 00:55:37 2022] PUT /identity/v3/projects/0f0baac91140411c9f2f1509031d3ff3/users/1ff9bd07b1f649ea8535d6cd8750bfec/roles/f08d6043261747e3b4763c744eef7dfb => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:55:37.289926 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-327ad51c-4d36-48db-a6e1-6a9202983e1c admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:37.290250 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 914/1812] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:55:37 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 00:55:37.294446 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1413111c-bba2-4fc3-89d4-8b6145626892 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:37.304365 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1413111c-bba2-4fc3-89d4-8b6145626892 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:37.322189 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:55:37.352264 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-327ad51c-4d36-48db-a6e1-6a9202983e1c admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:37.357118 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-930eefeb-7e76-40e0-85c5-3f3e89f9c244 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:37.357850 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-930eefeb-7e76-40e0-85c5-3f3e89f9c244 admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:37.358133 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-930eefeb-7e76-40e0-85c5-3f3e89f9c244 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:37.358406 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-930eefeb-7e76-40e0-85c5-3f3e89f9c244 admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:37.359054 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-930eefeb-7e76-40e0-85c5-3f3e89f9c244 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:37.360981 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-930eefeb-7e76-40e0-85c5-3f3e89f9c244 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:37.363349 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d75cfb5-6578-4645-84ba-b93cbafc7d9b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:37.364011 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d75cfb5-6578-4645-84ba-b93cbafc7d9b admin admin] REQUEST_METHOD: `PUT` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:37.364285 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d75cfb5-6578-4645-84ba-b93cbafc7d9b admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:37.364555 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d75cfb5-6578-4645-84ba-b93cbafc7d9b admin admin] PATH_INFO: `/v3/projects/13f4da7f3bd34f6eb435ae2bbe8bfa0d/users/a50e75501b1f4d7e91efd0c237d972d8/roles/f08d6043261747e3b4763c744eef7dfb` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:37.372413 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 899/1813] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:55:37 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 28 00:55:37.376746 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-930eefeb-7e76-40e0-85c5-3f3e89f9c244 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:37.381137 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d75cfb5-6578-4645-84ba-b93cbafc7d9b admin admin] RBAC: Authorizing `identity:create_grant(project_id=13f4da7f3bd34f6eb435ae2bbe8bfa0d, user_id=a50e75501b1f4d7e91efd0c237d972d8, role_id=f08d6043261747e3b4763c744eef7dfb)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:37.383786 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d75cfb5-6578-4645-84ba-b93cbafc7d9b admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:37.387309 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-930eefeb-7e76-40e0-85c5-3f3e89f9c244 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:37.393977 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a935549-448e-4134-9eb0-77324a02491f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:37.395182 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a935549-448e-4134-9eb0-77324a02491f admin admin] REQUEST_METHOD: `PUT` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:37.395506 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a935549-448e-4134-9eb0-77324a02491f admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:37.395778 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a935549-448e-4134-9eb0-77324a02491f admin admin] PATH_INFO: `/v3/projects/0f0baac91140411c9f2f1509031d3ff3/users/1ff9bd07b1f649ea8535d6cd8750bfec/roles/f08d6043261747e3b4763c744eef7dfb` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:37.397911 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a935549-448e-4134-9eb0-77324a02491f admin admin] RBAC: Authorizing `identity:create_grant(project_id=0f0baac91140411c9f2f1509031d3ff3, user_id=1ff9bd07b1f649ea8535d6cd8750bfec, role_id=f08d6043261747e3b4763c744eef7dfb)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:37.398684 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 915/1814] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 00:55:37 2022] PUT /identity/v3/projects/13f4da7f3bd34f6eb435ae2bbe8bfa0d/users/a50e75501b1f4d7e91efd0c237d972d8/roles/f08d6043261747e3b4763c744eef7dfb => generated 0 bytes in 105 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:55:37.402711 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d75cfb5-6578-4645-84ba-b93cbafc7d9b admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:37.402964 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a935549-448e-4134-9eb0-77324a02491f admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:37.414102 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 900/1815] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 00:55:37 2022] PUT /identity/v3/projects/0f0baac91140411c9f2f1509031d3ff3/users/1ff9bd07b1f649ea8535d6cd8750bfec/roles/f08d6043261747e3b4763c744eef7dfb => generated 0 bytes in 38 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:55:37.424880 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b74728d4-c7de-4500-b705-6b42a31cbc44 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:37.425202 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b74728d4-c7de-4500-b705-6b42a31cbc44 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:37.425541 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b74728d4-c7de-4500-b705-6b42a31cbc44 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:37.454977 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-b74728d4-c7de-4500-b705-6b42a31cbc44 None None] MFA Rules not processed for user `1ff9bd07b1f649ea8535d6cd8750bfec`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:55:37.456206 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-b74728d4-c7de-4500-b705-6b42a31cbc44 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:37.464594 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d75cfb5-6578-4645-84ba-b93cbafc7d9b admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:37.489168 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:55:37.524445 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62db9e49-f9fe-4660-a016-42a3fbb1b14a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:37.525356 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62db9e49-f9fe-4660-a016-42a3fbb1b14a admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:37.525633 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62db9e49-f9fe-4660-a016-42a3fbb1b14a admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:37.525901 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62db9e49-f9fe-4660-a016-42a3fbb1b14a admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:37.526531 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62db9e49-f9fe-4660-a016-42a3fbb1b14a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:37.528127 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62db9e49-f9fe-4660-a016-42a3fbb1b14a admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:37.528628 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 901/1816] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:55:37 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:55:37.540572 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 916/1817] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:55:37 2022] GET /identity/v3/roles => generated 1420 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 28 00:55:37.540715 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b74728d4-c7de-4500-b705-6b42a31cbc44 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:37.541518 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-b74728d4-c7de-4500-b705-6b42a31cbc44 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:37.542184 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62db9e49-f9fe-4660-a016-42a3fbb1b14a admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:37.552161 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62db9e49-f9fe-4660-a016-42a3fbb1b14a admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:37.557014 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7014f478-b6c1-4647-b55c-ddba7d1ea2e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:37.557651 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7014f478-b6c1-4647-b55c-ddba7d1ea2e6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:37.557928 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7014f478-b6c1-4647-b55c-ddba7d1ea2e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:37.558191 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7014f478-b6c1-4647-b55c-ddba7d1ea2e6 admin admin] PATH_INFO: `/v3/projects/13f4da7f3bd34f6eb435ae2bbe8bfa0d/users/a50e75501b1f4d7e91efd0c237d972d8/roles/f08d6043261747e3b4763c744eef7dfb` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:37.560055 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7014f478-b6c1-4647-b55c-ddba7d1ea2e6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=13f4da7f3bd34f6eb435ae2bbe8bfa0d, user_id=a50e75501b1f4d7e91efd0c237d972d8, role_id=f08d6043261747e3b4763c744eef7dfb)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:37.563007 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7014f478-b6c1-4647-b55c-ddba7d1ea2e6 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:37.573134 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 917/1818] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 00:55:37 2022] PUT /identity/v3/projects/13f4da7f3bd34f6eb435ae2bbe8bfa0d/users/a50e75501b1f4d7e91efd0c237d972d8/roles/f08d6043261747e3b4763c744eef7dfb => generated 0 bytes in 31 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 28 00:55:37.583654 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95029585-21ce-42bf-9ad8-ad6d25e8ba82 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:37.583943 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95029585-21ce-42bf-9ad8-ad6d25e8ba82 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:37.584201 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95029585-21ce-42bf-9ad8-ad6d25e8ba82 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:37.604145 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b74728d4-c7de-4500-b705-6b42a31cbc44 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:37.613283 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-95029585-21ce-42bf-9ad8-ad6d25e8ba82 None None] MFA Rules not processed for user `a50e75501b1f4d7e91efd0c237d972d8`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:55:37.614493 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-95029585-21ce-42bf-9ad8-ad6d25e8ba82 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:37.634425 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:55:37.654627 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15bf9a63-5da5-436c-a42a-4787a1c337a9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:37.655234 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15bf9a63-5da5-436c-a42a-4787a1c337a9 service cinder] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:37.655508 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15bf9a63-5da5-436c-a42a-4787a1c337a9 service cinder] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:37.655775 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15bf9a63-5da5-436c-a42a-4787a1c337a9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:37.682898 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 918/1819] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:55:37 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 28 00:55:37.694057 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95029585-21ce-42bf-9ad8-ad6d25e8ba82 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:37.695462 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-95029585-21ce-42bf-9ad8-ad6d25e8ba82 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:37.707450 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95029585-21ce-42bf-9ad8-ad6d25e8ba82 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:37.711934 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4b3a6a2-644b-461e-abec-7cc82896f3ff service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:37.713060 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4b3a6a2-644b-461e-abec-7cc82896f3ff service cinder] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:37.713344 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4b3a6a2-644b-461e-abec-7cc82896f3ff service cinder] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:37.713616 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4b3a6a2-644b-461e-abec-7cc82896f3ff service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:37.725882 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15bf9a63-5da5-436c-a42a-4787a1c337a9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:37.726391 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4b3a6a2-644b-461e-abec-7cc82896f3ff service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:37.727348 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15bf9a63-5da5-436c-a42a-4787a1c337a9 service cinder] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:37.727883 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4b3a6a2-644b-461e-abec-7cc82896f3ff service cinder] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:37.738564 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 919/1820] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:55:37 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:55:37.739223 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 902/1821] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:55:37 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 200 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:55:37.867889 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4b3a6a2-644b-461e-abec-7cc82896f3ff service cinder] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:37.879306 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4b3a6a2-644b-461e-abec-7cc82896f3ff service cinder] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:37.884775 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe8c843e-159c-448f-a936-b641305199a2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:37.885633 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe8c843e-159c-448f-a936-b641305199a2 service cinder] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:37.886267 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe8c843e-159c-448f-a936-b641305199a2 service cinder] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:37.886746 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe8c843e-159c-448f-a936-b641305199a2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:37.913330 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15bf9a63-5da5-436c-a42a-4787a1c337a9 service cinder] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:37.962188 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:55:37.975251 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe8c843e-159c-448f-a936-b641305199a2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:37.976931 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe8c843e-159c-448f-a936-b641305199a2 service cinder] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:38.010601 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 920/1822] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:55:37 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 143 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:55:38.012603 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15bf9a63-5da5-436c-a42a-4787a1c337a9 service cinder] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:38.017802 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3570b0ba-5fa7-48f9-99d3-baa90c816267 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:38.033741 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3570b0ba-5fa7-48f9-99d3-baa90c816267 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:38.034167 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3570b0ba-5fa7-48f9-99d3-baa90c816267 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:38.034611 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3570b0ba-5fa7-48f9-99d3-baa90c816267 admin admin] PATH_INFO: `/v3/users/432e8a2698dd478d9a04c2e0ff2ac458` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:38.036414 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3570b0ba-5fa7-48f9-99d3-baa90c816267 admin admin] RBAC: Authorizing `identity:delete_user(user_id=432e8a2698dd478d9a04c2e0ff2ac458)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:38.038705 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3570b0ba-5fa7-48f9-99d3-baa90c816267 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:38.100666 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-3570b0ba-5fa7-48f9-99d3-baa90c816267 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '432e8a2698dd478d9a04c2e0ff2ac458'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:38.123626 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-3570b0ba-5fa7-48f9-99d3-baa90c816267 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '432e8a2698dd478d9a04c2e0ff2ac458'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:38.136682 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-3570b0ba-5fa7-48f9-99d3-baa90c816267 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '432e8a2698dd478d9a04c2e0ff2ac458'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:38.147191 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-3570b0ba-5fa7-48f9-99d3-baa90c816267 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '432e8a2698dd478d9a04c2e0ff2ac458'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:38.147995 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-3570b0ba-5fa7-48f9-99d3-baa90c816267 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '432e8a2698dd478d9a04c2e0ff2ac458'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:38.159127 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 903/1823] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 00:55:37 2022] DELETE /identity/v3/users/432e8a2698dd478d9a04c2e0ff2ac458 => generated 0 bytes in 246 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 28 00:55:38.163841 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-307025ee-934a-4055-b0f4-175bf4e1a9ec None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:38.164326 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-307025ee-934a-4055-b0f4-175bf4e1a9ec None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:38.164443 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-307025ee-934a-4055-b0f4-175bf4e1a9ec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:38.201879 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-307025ee-934a-4055-b0f4-175bf4e1a9ec None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:55:38.204471 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-307025ee-934a-4055-b0f4-175bf4e1a9ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:38.322312 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 921/1824] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:55:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:55:38.334665 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3570b0ba-5fa7-48f9-99d3-baa90c816267 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:38.336211 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-3570b0ba-5fa7-48f9-99d3-baa90c816267 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:38.377275 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-307025ee-934a-4055-b0f4-175bf4e1a9ec None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:38.378704 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-307025ee-934a-4055-b0f4-175bf4e1a9ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:38.391579 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-307025ee-934a-4055-b0f4-175bf4e1a9ec None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:38.396218 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9024ff89-1c51-4374-a02e-b09ea51bbb96 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:38.396864 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9024ff89-1c51-4374-a02e-b09ea51bbb96 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:38.397151 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9024ff89-1c51-4374-a02e-b09ea51bbb96 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:38.397418 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9024ff89-1c51-4374-a02e-b09ea51bbb96 admin admin] PATH_INFO: `/v3/users/434692f2bda14675b9d53756453cb7b9` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:38.398561 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9024ff89-1c51-4374-a02e-b09ea51bbb96 admin admin] RBAC: Authorizing `identity:delete_user(user_id=434692f2bda14675b9d53756453cb7b9)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:38.400512 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9024ff89-1c51-4374-a02e-b09ea51bbb96 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:38.403872 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3570b0ba-5fa7-48f9-99d3-baa90c816267 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:38.407868 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77cac9dd-4458-4f1b-8f06-4bc3fdf847fc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:38.408482 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77cac9dd-4458-4f1b-8f06-4bc3fdf847fc service neutron] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:38.408878 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77cac9dd-4458-4f1b-8f06-4bc3fdf847fc service neutron] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:38.409200 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77cac9dd-4458-4f1b-8f06-4bc3fdf847fc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:38.418885 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77cac9dd-4458-4f1b-8f06-4bc3fdf847fc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:38.420350 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77cac9dd-4458-4f1b-8f06-4bc3fdf847fc service neutron] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:38.430533 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 904/1825] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:55:38 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:55:38.443192 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:55:38.450970 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-9024ff89-1c51-4374-a02e-b09ea51bbb96 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '434692f2bda14675b9d53756453cb7b9'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:38.451684 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-9024ff89-1c51-4374-a02e-b09ea51bbb96 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '434692f2bda14675b9d53756453cb7b9'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:38.459925 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-9024ff89-1c51-4374-a02e-b09ea51bbb96 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '434692f2bda14675b9d53756453cb7b9'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:38.467567 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-9024ff89-1c51-4374-a02e-b09ea51bbb96 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '434692f2bda14675b9d53756453cb7b9'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:38.469000 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-9024ff89-1c51-4374-a02e-b09ea51bbb96 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '434692f2bda14675b9d53756453cb7b9'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:38.488384 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 922/1826] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 00:55:38 2022] DELETE /identity/v3/users/434692f2bda14675b9d53756453cb7b9 => generated 0 bytes in 112 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 28 00:55:38.492992 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88711bd7-9df5-47fc-a315-4e253af3aadd None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:38.493385 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88711bd7-9df5-47fc-a315-4e253af3aadd None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:38.493663 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88711bd7-9df5-47fc-a315-4e253af3aadd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:38.512585 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-88711bd7-9df5-47fc-a315-4e253af3aadd None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:55:38.513848 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-88711bd7-9df5-47fc-a315-4e253af3aadd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:38.560522 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:55:38.577277 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 905/1827] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:55:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:55:38.597953 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9024ff89-1c51-4374-a02e-b09ea51bbb96 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:38.600227 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-9024ff89-1c51-4374-a02e-b09ea51bbb96 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:38.660839 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9024ff89-1c51-4374-a02e-b09ea51bbb96 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:38.664501 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d325804-99b4-424d-850e-68d1bda3a910 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:38.665146 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d325804-99b4-424d-850e-68d1bda3a910 service neutron] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:38.665426 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d325804-99b4-424d-850e-68d1bda3a910 service neutron] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:38.665693 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d325804-99b4-424d-850e-68d1bda3a910 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:38.674892 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d325804-99b4-424d-850e-68d1bda3a910 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:38.676334 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d325804-99b4-424d-850e-68d1bda3a910 service neutron] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:38.686024 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 923/1828] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:55:38 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:55:38.709151 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88711bd7-9df5-47fc-a315-4e253af3aadd None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:38.710735 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-88711bd7-9df5-47fc-a315-4e253af3aadd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:38.723952 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88711bd7-9df5-47fc-a315-4e253af3aadd None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:38.729253 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-054e55b6-d7f4-43cc-a1f5-a1dc6f729554 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:38.730063 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-054e55b6-d7f4-43cc-a1f5-a1dc6f729554 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:38.730501 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-054e55b6-d7f4-43cc-a1f5-a1dc6f729554 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:38.731254 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-054e55b6-d7f4-43cc-a1f5-a1dc6f729554 admin admin] PATH_INFO: `/v3/projects/0ec600f2e5e44dcf840e990f26c7d49f` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:38.732483 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-054e55b6-d7f4-43cc-a1f5-a1dc6f729554 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0ec600f2e5e44dcf840e990f26c7d49f)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:38.734671 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-054e55b6-d7f4-43cc-a1f5-a1dc6f729554 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:38.787860 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-054e55b6-d7f4-43cc-a1f5-a1dc6f729554 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0ec600f2e5e44dcf840e990f26c7d49f', 'request_id': 'req-054e55b6-d7f4-43cc-a1f5-a1dc6f729554'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:38.794053 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-054e55b6-d7f4-43cc-a1f5-a1dc6f729554 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0ec600f2e5e44dcf840e990f26c7d49f', 'request_id': 'req-054e55b6-d7f4-43cc-a1f5-a1dc6f729554'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:38.808907 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-054e55b6-d7f4-43cc-a1f5-a1dc6f729554 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0ec600f2e5e44dcf840e990f26c7d49f', 'request_id': 'req-054e55b6-d7f4-43cc-a1f5-a1dc6f729554'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:38.829698 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-054e55b6-d7f4-43cc-a1f5-a1dc6f729554 admin admin] The token cache is being invalidate because project 0ec600f2e5e44dcf840e990f26c7d49f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81507) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 00:55:38.829993 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-054e55b6-d7f4-43cc-a1f5-a1dc6f729554 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:38.830685 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-054e55b6-d7f4-43cc-a1f5-a1dc6f729554 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:38.832717 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 906/1829] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 00:55:38 2022] DELETE /identity/v3/projects/0ec600f2e5e44dcf840e990f26c7d49f => generated 0 bytes in 124 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:55:38.837424 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d325804-99b4-424d-850e-68d1bda3a910 service neutron] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:38.854416 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d325804-99b4-424d-850e-68d1bda3a910 service neutron] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:38.929012 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7768c64-2b75-480d-aa8c-ab3e3d635057 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:38.929660 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7768c64-2b75-480d-aa8c-ab3e3d635057 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:38.929943 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7768c64-2b75-480d-aa8c-ab3e3d635057 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:38.930219 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7768c64-2b75-480d-aa8c-ab3e3d635057 admin admin] PATH_INFO: `/v3/users/a50e75501b1f4d7e91efd0c237d972d8` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:38.931363 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7768c64-2b75-480d-aa8c-ab3e3d635057 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a50e75501b1f4d7e91efd0c237d972d8)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:38.933341 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7768c64-2b75-480d-aa8c-ab3e3d635057 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:38.991725 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-054e55b6-d7f4-43cc-a1f5-a1dc6f729554 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:38.994161 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-054e55b6-d7f4-43cc-a1f5-a1dc6f729554 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:38.995002 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-e7768c64-2b75-480d-aa8c-ab3e3d635057 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a50e75501b1f4d7e91efd0c237d972d8'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:38.995774 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-e7768c64-2b75-480d-aa8c-ab3e3d635057 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a50e75501b1f4d7e91efd0c237d972d8'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:39.007629 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-e7768c64-2b75-480d-aa8c-ab3e3d635057 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a50e75501b1f4d7e91efd0c237d972d8'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:39.016174 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-054e55b6-d7f4-43cc-a1f5-a1dc6f729554 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:39.020826 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:DELETE=13,keystone:SELECT=85,keystone:INSERT=2 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:55:39.025124 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-e7768c64-2b75-480d-aa8c-ab3e3d635057 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a50e75501b1f4d7e91efd0c237d972d8'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:39.026527 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b5edcce-e5bf-4523-8517-2869a2ce493a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:39.028414 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-e7768c64-2b75-480d-aa8c-ab3e3d635057 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a50e75501b1f4d7e91efd0c237d972d8'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:39.029953 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b5edcce-e5bf-4523-8517-2869a2ce493a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:39.030336 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b5edcce-e5bf-4523-8517-2869a2ce493a admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:39.030696 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b5edcce-e5bf-4523-8517-2869a2ce493a admin admin] PATH_INFO: `/v3/projects/a91de8ea5f65402d975e559d676388d9` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:39.032987 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b5edcce-e5bf-4523-8517-2869a2ce493a admin admin] RBAC: Authorizing `identity:delete_project(project_id=a91de8ea5f65402d975e559d676388d9)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:39.045029 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b5edcce-e5bf-4523-8517-2869a2ce493a admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:39.064222 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 924/1830] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 00:55:38 2022] DELETE /identity/v3/users/a50e75501b1f4d7e91efd0c237d972d8 => generated 0 bytes in 227 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:55:39.132535 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-2b5edcce-e5bf-4523-8517-2869a2ce493a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a91de8ea5f65402d975e559d676388d9', 'request_id': 'req-2b5edcce-e5bf-4523-8517-2869a2ce493a'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:39.139309 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-2b5edcce-e5bf-4523-8517-2869a2ce493a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a91de8ea5f65402d975e559d676388d9', 'request_id': 'req-2b5edcce-e5bf-4523-8517-2869a2ce493a'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:39.148683 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-2b5edcce-e5bf-4523-8517-2869a2ce493a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a91de8ea5f65402d975e559d676388d9', 'request_id': 'req-2b5edcce-e5bf-4523-8517-2869a2ce493a'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:39.174150 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-2b5edcce-e5bf-4523-8517-2869a2ce493a admin admin] The token cache is being invalidate because project a91de8ea5f65402d975e559d676388d9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81507) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 00:55:39.174451 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-2b5edcce-e5bf-4523-8517-2869a2ce493a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:39.176528 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-2b5edcce-e5bf-4523-8517-2869a2ce493a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:39.181873 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 907/1831] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 00:55:38 2022] DELETE /identity/v3/projects/a91de8ea5f65402d975e559d676388d9 => generated 0 bytes in 191 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 28 00:55:39.189497 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7768c64-2b75-480d-aa8c-ab3e3d635057 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:39.193667 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-e7768c64-2b75-480d-aa8c-ab3e3d635057 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:39.309399 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7768c64-2b75-480d-aa8c-ab3e3d635057 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:39.399615 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-473c41e4-2681-4ad0-a834-c102e4495f02 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:39.400367 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-473c41e4-2681-4ad0-a834-c102e4495f02 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:39.400529 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-473c41e4-2681-4ad0-a834-c102e4495f02 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:39.400913 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-473c41e4-2681-4ad0-a834-c102e4495f02 admin admin] PATH_INFO: `/v3/users/1ff9bd07b1f649ea8535d6cd8750bfec` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:39.401994 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-473c41e4-2681-4ad0-a834-c102e4495f02 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1ff9bd07b1f649ea8535d6cd8750bfec)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:39.403953 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-473c41e4-2681-4ad0-a834-c102e4495f02 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:39.464640 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-473c41e4-2681-4ad0-a834-c102e4495f02 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1ff9bd07b1f649ea8535d6cd8750bfec'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:39.465415 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-473c41e4-2681-4ad0-a834-c102e4495f02 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1ff9bd07b1f649ea8535d6cd8750bfec'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:39.474158 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-473c41e4-2681-4ad0-a834-c102e4495f02 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1ff9bd07b1f649ea8535d6cd8750bfec'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:39.481937 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-473c41e4-2681-4ad0-a834-c102e4495f02 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1ff9bd07b1f649ea8535d6cd8750bfec'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:39.482694 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-473c41e4-2681-4ad0-a834-c102e4495f02 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1ff9bd07b1f649ea8535d6cd8750bfec'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:39.504401 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 925/1832] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 00:55:39 2022] DELETE /identity/v3/users/1ff9bd07b1f649ea8535d6cd8750bfec => generated 0 bytes in 319 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:55:39.632590 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b5edcce-e5bf-4523-8517-2869a2ce493a admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:39.635907 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-2b5edcce-e5bf-4523-8517-2869a2ce493a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:39.677672 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b5edcce-e5bf-4523-8517-2869a2ce493a admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:39.756255 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76b5d6a0-0d85-47d4-8344-f2168ff9dd1a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:39.757062 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76b5d6a0-0d85-47d4-8344-f2168ff9dd1a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:39.757334 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76b5d6a0-0d85-47d4-8344-f2168ff9dd1a admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:39.757630 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76b5d6a0-0d85-47d4-8344-f2168ff9dd1a admin admin] PATH_INFO: `/v3/projects/13f4da7f3bd34f6eb435ae2bbe8bfa0d` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:39.758824 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76b5d6a0-0d85-47d4-8344-f2168ff9dd1a admin admin] RBAC: Authorizing `identity:delete_project(project_id=13f4da7f3bd34f6eb435ae2bbe8bfa0d)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:39.760770 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76b5d6a0-0d85-47d4-8344-f2168ff9dd1a admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:39.763311 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:55:39.798103 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-473c41e4-2681-4ad0-a834-c102e4495f02 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:39.800286 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-473c41e4-2681-4ad0-a834-c102e4495f02 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:39.812284 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:55:39.814364 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-76b5d6a0-0d85-47d4-8344-f2168ff9dd1a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '13f4da7f3bd34f6eb435ae2bbe8bfa0d', 'request_id': 'req-76b5d6a0-0d85-47d4-8344-f2168ff9dd1a'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:39.815918 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-473c41e4-2681-4ad0-a834-c102e4495f02 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:39.821017 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-76b5d6a0-0d85-47d4-8344-f2168ff9dd1a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '13f4da7f3bd34f6eb435ae2bbe8bfa0d', 'request_id': 'req-76b5d6a0-0d85-47d4-8344-f2168ff9dd1a'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:39.830563 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-76b5d6a0-0d85-47d4-8344-f2168ff9dd1a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '13f4da7f3bd34f6eb435ae2bbe8bfa0d', 'request_id': 'req-76b5d6a0-0d85-47d4-8344-f2168ff9dd1a'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:39.852711 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-76b5d6a0-0d85-47d4-8344-f2168ff9dd1a admin admin] The token cache is being invalidate because project 13f4da7f3bd34f6eb435ae2bbe8bfa0d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81507) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 00:55:39.853458 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-76b5d6a0-0d85-47d4-8344-f2168ff9dd1a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:39.854067 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-76b5d6a0-0d85-47d4-8344-f2168ff9dd1a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:39.856061 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 908/1833] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 00:55:39 2022] DELETE /identity/v3/projects/13f4da7f3bd34f6eb435ae2bbe8bfa0d => generated 0 bytes in 224 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:55:39.860764 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb3cb199-e46b-4260-a7a2-698626262c67 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:39.861100 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb3cb199-e46b-4260-a7a2-698626262c67 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:39.861454 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb3cb199-e46b-4260-a7a2-698626262c67 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:39.884543 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-bb3cb199-e46b-4260-a7a2-698626262c67 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:55:39.886034 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-bb3cb199-e46b-4260-a7a2-698626262c67 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:39.909238 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df967b98-7fcd-46b9-b3ea-e2ad7591dcdb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:39.910556 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df967b98-7fcd-46b9-b3ea-e2ad7591dcdb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:39.911151 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df967b98-7fcd-46b9-b3ea-e2ad7591dcdb admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:39.911626 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df967b98-7fcd-46b9-b3ea-e2ad7591dcdb admin admin] PATH_INFO: `/v3/projects/0f0baac91140411c9f2f1509031d3ff3` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:39.913296 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df967b98-7fcd-46b9-b3ea-e2ad7591dcdb admin admin] RBAC: Authorizing `identity:delete_project(project_id=0f0baac91140411c9f2f1509031d3ff3)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:39.915992 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df967b98-7fcd-46b9-b3ea-e2ad7591dcdb admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:39.969133 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 909/1834] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:55:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:55:39.972861 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-df967b98-7fcd-46b9-b3ea-e2ad7591dcdb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0f0baac91140411c9f2f1509031d3ff3', 'request_id': 'req-df967b98-7fcd-46b9-b3ea-e2ad7591dcdb'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:39.974962 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a0fa1b-d35a-4534-8ba8-ca8af7067c25 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:39.975382 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a0fa1b-d35a-4534-8ba8-ca8af7067c25 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:39.975755 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a0fa1b-d35a-4534-8ba8-ca8af7067c25 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:39.981662 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-df967b98-7fcd-46b9-b3ea-e2ad7591dcdb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0f0baac91140411c9f2f1509031d3ff3', 'request_id': 'req-df967b98-7fcd-46b9-b3ea-e2ad7591dcdb'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:39.995652 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-df967b98-7fcd-46b9-b3ea-e2ad7591dcdb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0f0baac91140411c9f2f1509031d3ff3', 'request_id': 'req-df967b98-7fcd-46b9-b3ea-e2ad7591dcdb'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:40.001856 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-80a0fa1b-d35a-4534-8ba8-ca8af7067c25 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:55:40.003363 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-80a0fa1b-d35a-4534-8ba8-ca8af7067c25 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:40.011112 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-df967b98-7fcd-46b9-b3ea-e2ad7591dcdb admin admin] The token cache is being invalidate because project 0f0baac91140411c9f2f1509031d3ff3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81508) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 00:55:40.011612 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-df967b98-7fcd-46b9-b3ea-e2ad7591dcdb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:40.012532 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-df967b98-7fcd-46b9-b3ea-e2ad7591dcdb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:40.017725 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 926/1835] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 00:55:39 2022] DELETE /identity/v3/projects/0f0baac91140411c9f2f1509031d3ff3 => generated 0 bytes in 218 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:55:40.019503 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5977c34b-e4ad-4e52-b1db-498a2fee7ecd None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:40.020003 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5977c34b-e4ad-4e52-b1db-498a2fee7ecd None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:40.020428 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5977c34b-e4ad-4e52-b1db-498a2fee7ecd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:40.043082 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-5977c34b-e4ad-4e52-b1db-498a2fee7ecd None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:55:40.044898 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-5977c34b-e4ad-4e52-b1db-498a2fee7ecd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:40.124042 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 910/1836] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:55:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 150 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 28 00:55:40.129593 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80a0fa1b-d35a-4534-8ba8-ca8af7067c25 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:40.152023 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80a0fa1b-d35a-4534-8ba8-ca8af7067c25 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:40.158615 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d469ac9-60ce-4416-a866-37f5ca900a69 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:40.159259 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d469ac9-60ce-4416-a866-37f5ca900a69 admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:40.159552 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d469ac9-60ce-4416-a866-37f5ca900a69 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:40.159812 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d469ac9-60ce-4416-a866-37f5ca900a69 admin admin] PATH_INFO: `/v3/domains` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:40.160484 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d469ac9-60ce-4416-a866-37f5ca900a69 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:55:40.160787 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d469ac9-60ce-4416-a866-37f5ca900a69 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:40.163067 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d469ac9-60ce-4416-a866-37f5ca900a69 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:40.169379 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 927/1837] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:55:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:55:40.184837 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-092c3559-d698-4c36-8577-a09a1a58a14f None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:40.186371 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-092c3559-d698-4c36-8577-a09a1a58a14f None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:40.186685 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-092c3559-d698-4c36-8577-a09a1a58a14f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:40.199658 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 911/1838] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Aug 28 00:55:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 67 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:55:40.201159 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d469ac9-60ce-4416-a866-37f5ca900a69 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:40.217694 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d469ac9-60ce-4416-a866-37f5ca900a69 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:40.235682 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70f74e9f-88a3-4d9c-a283-3333ccacc5a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:40.236487 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70f74e9f-88a3-4d9c-a283-3333ccacc5a7 admin admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:40.236868 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70f74e9f-88a3-4d9c-a283-3333ccacc5a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:40.237264 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70f74e9f-88a3-4d9c-a283-3333ccacc5a7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:40.238108 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70f74e9f-88a3-4d9c-a283-3333ccacc5a7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:40.240432 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70f74e9f-88a3-4d9c-a283-3333ccacc5a7 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:40.278216 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-092c3559-d698-4c36-8577-a09a1a58a14f None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:55:40.281058 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-092c3559-d698-4c36-8577-a09a1a58a14f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:40.301039 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 928/1839] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:55:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:55:40.307329 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-092c3559-d698-4c36-8577-a09a1a58a14f None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:40.319974 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-092c3559-d698-4c36-8577-a09a1a58a14f None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:40.330874 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:55:40.341479 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-322f744b-ee95-495a-ae58-f33ec1d1581c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:40.342161 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-322f744b-ee95-495a-ae58-f33ec1d1581c admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:40.342375 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-322f744b-ee95-495a-ae58-f33ec1d1581c admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:40.342643 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-322f744b-ee95-495a-ae58-f33ec1d1581c admin admin] PATH_INFO: `/v3/domains` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:40.343589 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-322f744b-ee95-495a-ae58-f33ec1d1581c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:55:40.343894 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-322f744b-ee95-495a-ae58-f33ec1d1581c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:40.347154 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-322f744b-ee95-495a-ae58-f33ec1d1581c admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:40.358888 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 912/1840] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 00:55:40 2022] POST /identity/v3/projects => generated 407 bytes in 157 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 28 00:55:40.361534 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70f74e9f-88a3-4d9c-a283-3333ccacc5a7 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:40.376126 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 929/1841] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Aug 28 00:55:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 70 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 28 00:55:40.380578 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-322f744b-ee95-495a-ae58-f33ec1d1581c admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:40.405186 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-322f744b-ee95-495a-ae58-f33ec1d1581c admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:40.438232 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:55:40.553336 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31742401-0743-4626-b3f6-b9eb4afe6362 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:40.554285 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31742401-0743-4626-b3f6-b9eb4afe6362 admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:40.554717 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31742401-0743-4626-b3f6-b9eb4afe6362 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:40.555153 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31742401-0743-4626-b3f6-b9eb4afe6362 admin admin] PATH_INFO: `/v3/projects` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:40.555945 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31742401-0743-4626-b3f6-b9eb4afe6362 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:40.558962 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31742401-0743-4626-b3f6-b9eb4afe6362 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:40.568238 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70f74e9f-88a3-4d9c-a283-3333ccacc5a7 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:40.589141 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d19b042-daac-4a4b-b6a8-322567158b46 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:40.591853 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d19b042-daac-4a4b-b6a8-322567158b46 admin admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:40.592665 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d19b042-daac-4a4b-b6a8-322567158b46 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:40.594218 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d19b042-daac-4a4b-b6a8-322567158b46 admin admin] PATH_INFO: `/v3/users` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:40.597766 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d19b042-daac-4a4b-b6a8-322567158b46 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:40.606040 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d19b042-daac-4a4b-b6a8-322567158b46 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:40.621534 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 930/1842] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 00:55:40 2022] POST /identity/v3/projects => generated 373 bytes in 241 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 28 00:55:40.626228 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31742401-0743-4626-b3f6-b9eb4afe6362 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:40.657510 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 913/1843] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 00:55:40 2022] POST /identity/v3/users => generated 304 bytes in 296 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:55:40.686953 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d19b042-daac-4a4b-b6a8-322567158b46 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:40.725912 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d19b042-daac-4a4b-b6a8-322567158b46 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:40.766013 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31742401-0743-4626-b3f6-b9eb4afe6362 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:40.777651 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d44cb7f0-c83d-4393-9756-88cefb8ac5ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:40.778283 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d44cb7f0-c83d-4393-9756-88cefb8ac5ff admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:40.778559 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d44cb7f0-c83d-4393-9756-88cefb8ac5ff admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:40.778829 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d44cb7f0-c83d-4393-9756-88cefb8ac5ff admin admin] PATH_INFO: `/v3/users` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:40.782135 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d44cb7f0-c83d-4393-9756-88cefb8ac5ff admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:40.784463 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d44cb7f0-c83d-4393-9756-88cefb8ac5ff admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:40.845980 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 931/1844] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 00:55:40 2022] POST /identity/v3/users => generated 287 bytes in 220 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 28 00:55:40.846794 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d44cb7f0-c83d-4393-9756-88cefb8ac5ff admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:40.874107 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d44cb7f0-c83d-4393-9756-88cefb8ac5ff admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:40.889238 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b20830a1-c696-44f6-b765-ac981af19d10 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:40.889875 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b20830a1-c696-44f6-b765-ac981af19d10 admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:40.890155 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b20830a1-c696-44f6-b765-ac981af19d10 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:40.890480 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b20830a1-c696-44f6-b765-ac981af19d10 admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:40.891135 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b20830a1-c696-44f6-b765-ac981af19d10 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:40.892756 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b20830a1-c696-44f6-b765-ac981af19d10 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:40.898234 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82793f40-37e1-41e1-9df2-5255780e8a7b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:40.901910 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82793f40-37e1-41e1-9df2-5255780e8a7b admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:40.902185 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82793f40-37e1-41e1-9df2-5255780e8a7b admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:40.904930 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82793f40-37e1-41e1-9df2-5255780e8a7b admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:40.905665 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82793f40-37e1-41e1-9df2-5255780e8a7b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:40.911124 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82793f40-37e1-41e1-9df2-5255780e8a7b admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:40.929761 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 932/1845] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:55:40 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 00:55:40.930642 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b20830a1-c696-44f6-b765-ac981af19d10 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:40.938006 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 914/1846] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:55:40 2022] GET /identity/v3/roles => generated 1420 bytes in 276 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 00:55:40.942044 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82793f40-37e1-41e1-9df2-5255780e8a7b admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:40.942731 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b20830a1-c696-44f6-b765-ac981af19d10 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:40.949824 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3ec5ec9-818e-4616-8b25-443e3acb832d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:40.950458 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3ec5ec9-818e-4616-8b25-443e3acb832d admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:40.950734 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3ec5ec9-818e-4616-8b25-443e3acb832d admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:40.951000 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3ec5ec9-818e-4616-8b25-443e3acb832d admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:40.952280 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3ec5ec9-818e-4616-8b25-443e3acb832d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:40.954046 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3ec5ec9-818e-4616-8b25-443e3acb832d admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:40.965362 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82793f40-37e1-41e1-9df2-5255780e8a7b admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:40.972582 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f904d3c1-4702-4145-87d4-5f34728b7342 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:40.974061 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f904d3c1-4702-4145-87d4-5f34728b7342 admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:40.975231 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f904d3c1-4702-4145-87d4-5f34728b7342 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:40.975676 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f904d3c1-4702-4145-87d4-5f34728b7342 admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:40.976591 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f904d3c1-4702-4145-87d4-5f34728b7342 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:40.978811 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 933/1847] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:55:40 2022] GET /identity/v3/roles => generated 1420 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 00:55:40.979364 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f904d3c1-4702-4145-87d4-5f34728b7342 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:40.988204 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3ec5ec9-818e-4616-8b25-443e3acb832d admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:41.004026 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 915/1848] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:55:40 2022] GET /identity/v3/roles => generated 1420 bytes in 65 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 00:55:41.008303 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3ec5ec9-818e-4616-8b25-443e3acb832d admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:41.014277 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f904d3c1-4702-4145-87d4-5f34728b7342 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:41.024174 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-850d00f8-7ffc-490d-9a2f-66bbab6d9481 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:41.025343 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-850d00f8-7ffc-490d-9a2f-66bbab6d9481 admin admin] REQUEST_METHOD: `PUT` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:41.025646 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-850d00f8-7ffc-490d-9a2f-66bbab6d9481 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:41.025914 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-850d00f8-7ffc-490d-9a2f-66bbab6d9481 admin admin] PATH_INFO: `/v3/projects/72d44c3f93034acfa2eeb77cb85a66b9/users/8e0d0d02bcab491f98409142789d5212/roles/0487ef7e943e4701915cb8ce0f2899d7` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:41.029283 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f904d3c1-4702-4145-87d4-5f34728b7342 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:41.046448 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecd90142-3e41-451e-b607-a46dc74f2860 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:41.047301 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecd90142-3e41-451e-b607-a46dc74f2860 admin admin] REQUEST_METHOD: `PUT` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:41.047604 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecd90142-3e41-451e-b607-a46dc74f2860 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:41.047883 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecd90142-3e41-451e-b607-a46dc74f2860 admin admin] PATH_INFO: `/v3/projects/25f494d3d7de482cb6131101a0cee139/users/2c9ba24f2fc142aa95949aeb2a5551b0/roles/0487ef7e943e4701915cb8ce0f2899d7` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:41.070032 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-850d00f8-7ffc-490d-9a2f-66bbab6d9481 admin admin] RBAC: Authorizing `identity:create_grant(project_id=72d44c3f93034acfa2eeb77cb85a66b9, user_id=8e0d0d02bcab491f98409142789d5212, role_id=0487ef7e943e4701915cb8ce0f2899d7)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:41.077129 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-850d00f8-7ffc-490d-9a2f-66bbab6d9481 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:41.089463 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecd90142-3e41-451e-b607-a46dc74f2860 admin admin] RBAC: Authorizing `identity:create_grant(project_id=25f494d3d7de482cb6131101a0cee139, user_id=2c9ba24f2fc142aa95949aeb2a5551b0, role_id=0487ef7e943e4701915cb8ce0f2899d7)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:41.093373 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecd90142-3e41-451e-b607-a46dc74f2860 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:41.103584 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 934/1849] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 00:55:40 2022] PUT /identity/v3/projects/72d44c3f93034acfa2eeb77cb85a66b9/users/8e0d0d02bcab491f98409142789d5212/roles/0487ef7e943e4701915cb8ce0f2899d7 => generated 0 bytes in 118 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:55:41.118035 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9189b139-5ae1-4b15-a96e-30dee94161dc None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:41.118559 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9189b139-5ae1-4b15-a96e-30dee94161dc None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:41.119294 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9189b139-5ae1-4b15-a96e-30dee94161dc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:41.141774 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 916/1850] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 00:55:41 2022] PUT /identity/v3/projects/25f494d3d7de482cb6131101a0cee139/users/2c9ba24f2fc142aa95949aeb2a5551b0/roles/0487ef7e943e4701915cb8ce0f2899d7 => generated 0 bytes in 128 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 28 00:55:41.153663 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5018d378-e5a8-473f-b5d1-dbd02747b6d6 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:41.153949 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5018d378-e5a8-473f-b5d1-dbd02747b6d6 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:41.154297 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5018d378-e5a8-473f-b5d1-dbd02747b6d6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:41.169710 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-9189b139-5ae1-4b15-a96e-30dee94161dc None None] MFA Rules not processed for user `8e0d0d02bcab491f98409142789d5212`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:55:41.171367 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-9189b139-5ae1-4b15-a96e-30dee94161dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:41.216899 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-5018d378-e5a8-473f-b5d1-dbd02747b6d6 None None] MFA Rules not processed for user `2c9ba24f2fc142aa95949aeb2a5551b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:55:41.219501 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:55:41.221558 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-5018d378-e5a8-473f-b5d1-dbd02747b6d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:41.266325 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 935/1851] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:55:41 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 150 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:55:41.272052 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9189b139-5ae1-4b15-a96e-30dee94161dc None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:41.292663 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:55:41.338751 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 917/1852] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:55:41 2022] POST /identity/v3/auth/tokens => generated 3284 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 28 00:55:41.346007 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5018d378-e5a8-473f-b5d1-dbd02747b6d6 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:41.399632 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9189b139-5ae1-4b15-a96e-30dee94161dc None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:41.413964 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6d6aa3f-44c2-4a9e-88bf-9601b625c201 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:41.414720 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6d6aa3f-44c2-4a9e-88bf-9601b625c201 admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:41.415096 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6d6aa3f-44c2-4a9e-88bf-9601b625c201 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:41.415326 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6d6aa3f-44c2-4a9e-88bf-9601b625c201 admin admin] PATH_INFO: `/v3/projects` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:41.416628 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6d6aa3f-44c2-4a9e-88bf-9601b625c201 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:41.419781 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6d6aa3f-44c2-4a9e-88bf-9601b625c201 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:41.439821 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5018d378-e5a8-473f-b5d1-dbd02747b6d6 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:41.446182 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05b4345b-5a4c-4874-b529-f7568222aaed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:41.446819 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b4345b-5a4c-4874-b529-f7568222aaed admin admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:41.447156 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b4345b-5a4c-4874-b529-f7568222aaed admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:41.447437 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b4345b-5a4c-4874-b529-f7568222aaed admin admin] PATH_INFO: `/v3/projects` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:41.448547 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05b4345b-5a4c-4874-b529-f7568222aaed admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:41.451559 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05b4345b-5a4c-4874-b529-f7568222aaed admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:41.480726 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 936/1853] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 00:55:41 2022] POST /identity/v3/projects => generated 377 bytes in 210 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 28 00:55:41.485771 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6d6aa3f-44c2-4a9e-88bf-9601b625c201 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:41.505327 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 918/1854] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 00:55:41 2022] POST /identity/v3/projects => generated 407 bytes in 162 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 28 00:55:41.510125 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05b4345b-5a4c-4874-b529-f7568222aaed admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:41.567451 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6d6aa3f-44c2-4a9e-88bf-9601b625c201 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:41.652426 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7434ec3e-24bc-48ca-8bf1-a00f3dd4fe03 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:41.653100 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7434ec3e-24bc-48ca-8bf1-a00f3dd4fe03 admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:41.653379 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7434ec3e-24bc-48ca-8bf1-a00f3dd4fe03 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:41.653680 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7434ec3e-24bc-48ca-8bf1-a00f3dd4fe03 admin admin] PATH_INFO: `/v3/users` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:41.654821 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7434ec3e-24bc-48ca-8bf1-a00f3dd4fe03 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:41.657117 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7434ec3e-24bc-48ca-8bf1-a00f3dd4fe03 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:41.670318 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:55:41.688998 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 937/1855] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 00:55:41 2022] POST /identity/v3/users => generated 289 bytes in 204 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:55:41.690039 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05b4345b-5a4c-4874-b529-f7568222aaed admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:41.690510 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7434ec3e-24bc-48ca-8bf1-a00f3dd4fe03 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:41.696724 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6986cef0-a97f-4608-a405-f8254f931e0c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:41.697908 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6986cef0-a97f-4608-a405-f8254f931e0c admin admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:41.698540 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6986cef0-a97f-4608-a405-f8254f931e0c admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:41.702409 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7434ec3e-24bc-48ca-8bf1-a00f3dd4fe03 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:41.703750 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6986cef0-a97f-4608-a405-f8254f931e0c admin admin] PATH_INFO: `/v3/users` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:41.705245 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6986cef0-a97f-4608-a405-f8254f931e0c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:41.707983 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fe98765-1433-40c3-a5af-cc86c6877492 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:41.708609 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fe98765-1433-40c3-a5af-cc86c6877492 admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:41.708887 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fe98765-1433-40c3-a5af-cc86c6877492 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:41.709209 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fe98765-1433-40c3-a5af-cc86c6877492 admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:41.709857 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fe98765-1433-40c3-a5af-cc86c6877492 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:41.710329 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6986cef0-a97f-4608-a405-f8254f931e0c admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:41.711453 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fe98765-1433-40c3-a5af-cc86c6877492 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:41.724100 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 938/1856] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:55:41 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 00:55:41.728823 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fe98765-1433-40c3-a5af-cc86c6877492 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:41.743147 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fe98765-1433-40c3-a5af-cc86c6877492 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:41.751848 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b07924d6-2a4b-4a73-a69b-971535f84c89 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:41.752475 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b07924d6-2a4b-4a73-a69b-971535f84c89 admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:41.752902 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b07924d6-2a4b-4a73-a69b-971535f84c89 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:41.753228 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b07924d6-2a4b-4a73-a69b-971535f84c89 admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:41.754367 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b07924d6-2a4b-4a73-a69b-971535f84c89 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:41.757263 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b07924d6-2a4b-4a73-a69b-971535f84c89 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:41.758268 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 919/1857] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 00:55:41 2022] POST /identity/v3/users => generated 310 bytes in 249 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:55:41.764407 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6986cef0-a97f-4608-a405-f8254f931e0c admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:41.776855 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 939/1858] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:55:41 2022] GET /identity/v3/roles => generated 1420 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 00:55:41.778554 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b07924d6-2a4b-4a73-a69b-971535f84c89 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:41.789430 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b07924d6-2a4b-4a73-a69b-971535f84c89 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:41.794700 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6986cef0-a97f-4608-a405-f8254f931e0c admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:41.795923 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c36cd56-60bf-4a44-af59-56f023d77027 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:41.796828 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c36cd56-60bf-4a44-af59-56f023d77027 admin admin] REQUEST_METHOD: `PUT` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:41.796996 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c36cd56-60bf-4a44-af59-56f023d77027 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:41.798726 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c36cd56-60bf-4a44-af59-56f023d77027 admin admin] PATH_INFO: `/v3/projects/c062f3365c2f496290dd2f383776476c/users/c0d3efa702504e9da6a14065c0619ab3/roles/0487ef7e943e4701915cb8ce0f2899d7` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:41.810468 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feed95e9-3233-45ec-a120-5c23c09557b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:41.811812 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feed95e9-3233-45ec-a120-5c23c09557b3 admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:41.812358 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feed95e9-3233-45ec-a120-5c23c09557b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:41.812835 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feed95e9-3233-45ec-a120-5c23c09557b3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:41.814051 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feed95e9-3233-45ec-a120-5c23c09557b3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:41.816054 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feed95e9-3233-45ec-a120-5c23c09557b3 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:41.828514 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c36cd56-60bf-4a44-af59-56f023d77027 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c062f3365c2f496290dd2f383776476c, user_id=c0d3efa702504e9da6a14065c0619ab3, role_id=0487ef7e943e4701915cb8ce0f2899d7)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:41.831694 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c36cd56-60bf-4a44-af59-56f023d77027 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:41.833616 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 920/1859] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:55:41 2022] GET /identity/v3/roles => generated 1420 bytes in 70 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 00:55:41.839407 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feed95e9-3233-45ec-a120-5c23c09557b3 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:41.854977 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feed95e9-3233-45ec-a120-5c23c09557b3 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:41.863238 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 940/1860] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 00:55:41 2022] PUT /identity/v3/projects/c062f3365c2f496290dd2f383776476c/users/c0d3efa702504e9da6a14065c0619ab3/roles/0487ef7e943e4701915cb8ce0f2899d7 => generated 0 bytes in 86 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:55:41.875252 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2b281bf-bbbd-4f8c-a1fa-d5de45f035ea None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:41.875570 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2b281bf-bbbd-4f8c-a1fa-d5de45f035ea None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:41.875806 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2b281bf-bbbd-4f8c-a1fa-d5de45f035ea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:41.909411 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-d2b281bf-bbbd-4f8c-a1fa-d5de45f035ea None None] MFA Rules not processed for user `c0d3efa702504e9da6a14065c0619ab3`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:55:41.910940 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-d2b281bf-bbbd-4f8c-a1fa-d5de45f035ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:41.918223 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:55:41.937077 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57c1b101-71e9-4d71-9a54-88c327755b94 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:41.937856 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57c1b101-71e9-4d71-9a54-88c327755b94 admin admin] REQUEST_METHOD: `PUT` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:41.938133 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57c1b101-71e9-4d71-9a54-88c327755b94 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:41.938415 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57c1b101-71e9-4d71-9a54-88c327755b94 admin admin] PATH_INFO: `/v3/projects/9d823c42c76544638a09799689a218eb/users/e47fae5af1e94d9fb37a4bedbf7c0910/roles/f08d6043261747e3b4763c744eef7dfb` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:41.958168 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57c1b101-71e9-4d71-9a54-88c327755b94 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9d823c42c76544638a09799689a218eb, user_id=e47fae5af1e94d9fb37a4bedbf7c0910, role_id=f08d6043261747e3b4763c744eef7dfb)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:41.961534 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57c1b101-71e9-4d71-9a54-88c327755b94 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:41.983766 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 921/1861] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 00:55:41 2022] PUT /identity/v3/projects/9d823c42c76544638a09799689a218eb/users/e47fae5af1e94d9fb37a4bedbf7c0910/roles/f08d6043261747e3b4763c744eef7dfb => generated 0 bytes in 143 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:55:41.985442 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57c1b101-71e9-4d71-9a54-88c327755b94 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:42.019616 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 941/1862] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:55:41 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:55:42.021056 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2b281bf-bbbd-4f8c-a1fa-d5de45f035ea None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:42.036658 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2b281bf-bbbd-4f8c-a1fa-d5de45f035ea None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:42.142762 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57c1b101-71e9-4d71-9a54-88c327755b94 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:42.158954 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19854444-c549-4c2f-97d7-bb7931dbc8b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:42.159664 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19854444-c549-4c2f-97d7-bb7931dbc8b6 admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:42.159888 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19854444-c549-4c2f-97d7-bb7931dbc8b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:42.160158 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19854444-c549-4c2f-97d7-bb7931dbc8b6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:42.165243 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f6d13a2-3b71-4e9a-84e9-190bde38943a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:42.165880 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f6d13a2-3b71-4e9a-84e9-190bde38943a admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:42.166155 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f6d13a2-3b71-4e9a-84e9-190bde38943a admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:42.166424 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f6d13a2-3b71-4e9a-84e9-190bde38943a admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:42.167065 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f6d13a2-3b71-4e9a-84e9-190bde38943a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:42.168495 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19854444-c549-4c2f-97d7-bb7931dbc8b6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:42.168813 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f6d13a2-3b71-4e9a-84e9-190bde38943a admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:42.174646 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19854444-c549-4c2f-97d7-bb7931dbc8b6 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:42.195723 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 922/1863] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:55:41 2022] GET /identity/v3/roles => generated 1420 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 00:55:42.202935 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f6d13a2-3b71-4e9a-84e9-190bde38943a admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:42.222086 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f6d13a2-3b71-4e9a-84e9-190bde38943a admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:42.244657 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80889144-e143-45c8-8ea3-b3ed9eb425af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:42.245373 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80889144-e143-45c8-8ea3-b3ed9eb425af admin admin] REQUEST_METHOD: `PUT` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:42.245735 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80889144-e143-45c8-8ea3-b3ed9eb425af admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:42.245921 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80889144-e143-45c8-8ea3-b3ed9eb425af admin admin] PATH_INFO: `/v3/projects/9d823c42c76544638a09799689a218eb/users/e47fae5af1e94d9fb37a4bedbf7c0910/roles/f08d6043261747e3b4763c744eef7dfb` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:42.247929 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80889144-e143-45c8-8ea3-b3ed9eb425af admin admin] RBAC: Authorizing `identity:create_grant(project_id=9d823c42c76544638a09799689a218eb, user_id=e47fae5af1e94d9fb37a4bedbf7c0910, role_id=f08d6043261747e3b4763c744eef7dfb)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:42.250676 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80889144-e143-45c8-8ea3-b3ed9eb425af admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:42.252992 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 942/1864] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 00:55:42 2022] POST /identity/v3/projects => generated 375 bytes in 232 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:55:42.258731 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19854444-c549-4c2f-97d7-bb7931dbc8b6 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:42.282125 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 923/1865] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 00:55:42 2022] PUT /identity/v3/projects/9d823c42c76544638a09799689a218eb/users/e47fae5af1e94d9fb37a4bedbf7c0910/roles/f08d6043261747e3b4763c744eef7dfb => generated 0 bytes in 83 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:55:42.292851 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abb5b351-f222-4434-a6ec-a339d7fbf45a None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:42.293368 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abb5b351-f222-4434-a6ec-a339d7fbf45a None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:42.293802 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abb5b351-f222-4434-a6ec-a339d7fbf45a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:42.342922 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:55:42.355313 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-abb5b351-f222-4434-a6ec-a339d7fbf45a None None] MFA Rules not processed for user `e47fae5af1e94d9fb37a4bedbf7c0910`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:55:42.358072 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-abb5b351-f222-4434-a6ec-a339d7fbf45a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:42.447034 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19854444-c549-4c2f-97d7-bb7931dbc8b6 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:42.471848 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 924/1866] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:55:42 2022] POST /identity/v3/auth/tokens => generated 3351 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 28 00:55:42.482762 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abb5b351-f222-4434-a6ec-a339d7fbf45a None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:42.483999 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-abb5b351-f222-4434-a6ec-a339d7fbf45a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:42.552701 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deec0075-4ade-44c2-afcb-6283d3ff775a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:42.553226 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deec0075-4ade-44c2-afcb-6283d3ff775a admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:42.553944 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deec0075-4ade-44c2-afcb-6283d3ff775a admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:42.554297 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deec0075-4ade-44c2-afcb-6283d3ff775a admin admin] PATH_INFO: `/v3/users` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:42.555703 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-deec0075-4ade-44c2-afcb-6283d3ff775a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:42.558474 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-deec0075-4ade-44c2-afcb-6283d3ff775a admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:42.562994 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:55:42.593156 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 943/1867] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 00:55:42 2022] POST /identity/v3/users => generated 294 bytes in 334 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:55:42.595738 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deec0075-4ade-44c2-afcb-6283d3ff775a admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:42.604730 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abb5b351-f222-4434-a6ec-a339d7fbf45a None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:42.608644 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd1592cc-6d26-4a6f-8f0b-15bf6a536049 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:42.609362 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd1592cc-6d26-4a6f-8f0b-15bf6a536049 service cinder] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:42.609657 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd1592cc-6d26-4a6f-8f0b-15bf6a536049 service cinder] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:42.609976 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd1592cc-6d26-4a6f-8f0b-15bf6a536049 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:42.610695 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deec0075-4ade-44c2-afcb-6283d3ff775a admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:42.618094 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-295a4a91-7059-485b-b788-d99298a0f2ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:42.618871 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-295a4a91-7059-485b-b788-d99298a0f2ba admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:42.619310 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-295a4a91-7059-485b-b788-d99298a0f2ba admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:42.619655 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-295a4a91-7059-485b-b788-d99298a0f2ba admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:42.620507 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-295a4a91-7059-485b-b788-d99298a0f2ba admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:42.621251 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd1592cc-6d26-4a6f-8f0b-15bf6a536049 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:42.622285 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-295a4a91-7059-485b-b788-d99298a0f2ba admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:42.622756 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd1592cc-6d26-4a6f-8f0b-15bf6a536049 service cinder] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:42.635540 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 944/1868] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:55:42 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 00:55:42.636104 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 925/1869] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:55:42 2022] GET /identity/v3/auth/tokens => generated 3351 bytes in 156 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:55:42.642042 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-295a4a91-7059-485b-b788-d99298a0f2ba admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:42.653676 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-295a4a91-7059-485b-b788-d99298a0f2ba admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:42.659608 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-100d3b32-0032-4767-81d1-50faae3930b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:42.660315 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-100d3b32-0032-4767-81d1-50faae3930b6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:42.661277 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-100d3b32-0032-4767-81d1-50faae3930b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:42.661877 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-100d3b32-0032-4767-81d1-50faae3930b6 admin admin] PATH_INFO: `/v3/projects/dc992a50f7e74e77af91babe2d17f5ad/users/dd25ccae1bcc4099bddee8a1259317a5/roles/f08d6043261747e3b4763c744eef7dfb` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:42.688626 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-100d3b32-0032-4767-81d1-50faae3930b6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=dc992a50f7e74e77af91babe2d17f5ad, user_id=dd25ccae1bcc4099bddee8a1259317a5, role_id=f08d6043261747e3b4763c744eef7dfb)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:42.692107 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-100d3b32-0032-4767-81d1-50faae3930b6 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:42.725036 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 945/1870] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 00:55:42 2022] PUT /identity/v3/projects/dc992a50f7e74e77af91babe2d17f5ad/users/dd25ccae1bcc4099bddee8a1259317a5/roles/f08d6043261747e3b4763c744eef7dfb => generated 0 bytes in 86 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 28 00:55:42.728844 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd1592cc-6d26-4a6f-8f0b-15bf6a536049 service cinder] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:42.740455 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd1592cc-6d26-4a6f-8f0b-15bf6a536049 service cinder] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:42.831912 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-100d3b32-0032-4767-81d1-50faae3930b6 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:42.834503 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-100d3b32-0032-4767-81d1-50faae3930b6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:42.847810 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7adb28de-1f89-4a3d-899f-78e174e6e869 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:42.848648 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7adb28de-1f89-4a3d-899f-78e174e6e869 admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:42.848936 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7adb28de-1f89-4a3d-899f-78e174e6e869 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:42.849338 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7adb28de-1f89-4a3d-899f-78e174e6e869 admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:42.850574 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7adb28de-1f89-4a3d-899f-78e174e6e869 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:42.855777 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7adb28de-1f89-4a3d-899f-78e174e6e869 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:42.867156 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 926/1871] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:55:42 2022] GET /identity/v3/roles => generated 1420 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 00:55:42.871298 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7adb28de-1f89-4a3d-899f-78e174e6e869 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:42.883027 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7adb28de-1f89-4a3d-899f-78e174e6e869 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:42.888839 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57e780e3-4844-4275-a55e-e56387ba8615 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:42.889885 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e780e3-4844-4275-a55e-e56387ba8615 admin admin] REQUEST_METHOD: `PUT` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:42.890063 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e780e3-4844-4275-a55e-e56387ba8615 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:42.890366 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e780e3-4844-4275-a55e-e56387ba8615 admin admin] PATH_INFO: `/v3/projects/dc992a50f7e74e77af91babe2d17f5ad/users/dd25ccae1bcc4099bddee8a1259317a5/roles/f08d6043261747e3b4763c744eef7dfb` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:42.892425 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57e780e3-4844-4275-a55e-e56387ba8615 admin admin] RBAC: Authorizing `identity:create_grant(project_id=dc992a50f7e74e77af91babe2d17f5ad, user_id=dd25ccae1bcc4099bddee8a1259317a5, role_id=f08d6043261747e3b4763c744eef7dfb)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:42.895504 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57e780e3-4844-4275-a55e-e56387ba8615 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:42.909017 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 927/1872] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 00:55:42 2022] PUT /identity/v3/projects/dc992a50f7e74e77af91babe2d17f5ad/users/dd25ccae1bcc4099bddee8a1259317a5/roles/f08d6043261747e3b4763c744eef7dfb => generated 0 bytes in 37 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 28 00:55:42.909017 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57e780e3-4844-4275-a55e-e56387ba8615 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:42.911858 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-57e780e3-4844-4275-a55e-e56387ba8615 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:42.940213 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-100d3b32-0032-4767-81d1-50faae3930b6 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:42.976061 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:55:42.980695 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57e780e3-4844-4275-a55e-e56387ba8615 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:42.985106 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4588d972-cbbd-41a6-bef8-3bb5cb70edbf service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:42.985902 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4588d972-cbbd-41a6-bef8-3bb5cb70edbf service cinder] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:42.986182 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4588d972-cbbd-41a6-bef8-3bb5cb70edbf service cinder] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:42.986450 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4588d972-cbbd-41a6-bef8-3bb5cb70edbf service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:43.010281 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5f4dd57-d445-4609-89c2-921be85e7eca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:43.010967 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5f4dd57-d445-4609-89c2-921be85e7eca admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:43.011248 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5f4dd57-d445-4609-89c2-921be85e7eca admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:43.011520 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5f4dd57-d445-4609-89c2-921be85e7eca admin admin] PATH_INFO: `/v3/users/5ae53fa7d26844618db6b1ae8c887c89` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:43.012701 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5f4dd57-d445-4609-89c2-921be85e7eca admin admin] RBAC: Authorizing `identity:delete_user(user_id=5ae53fa7d26844618db6b1ae8c887c89)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:43.014722 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5f4dd57-d445-4609-89c2-921be85e7eca admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:43.052237 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4588d972-cbbd-41a6-bef8-3bb5cb70edbf service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:43.053793 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4588d972-cbbd-41a6-bef8-3bb5cb70edbf service cinder] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:43.064059 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 928/1873] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:55:42 2022] GET /identity/v3/auth/tokens => generated 3284 bytes in 156 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:55:43.067183 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-618764b3-dd83-4992-8c66-6a25fd022769 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:43.067992 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-618764b3-dd83-4992-8c66-6a25fd022769 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:43.068191 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-618764b3-dd83-4992-8c66-6a25fd022769 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:43.072509 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-a5f4dd57-d445-4609-89c2-921be85e7eca admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5ae53fa7d26844618db6b1ae8c887c89'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:43.073530 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-a5f4dd57-d445-4609-89c2-921be85e7eca admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5ae53fa7d26844618db6b1ae8c887c89'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:43.084294 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-a5f4dd57-d445-4609-89c2-921be85e7eca admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5ae53fa7d26844618db6b1ae8c887c89'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:43.093313 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-a5f4dd57-d445-4609-89c2-921be85e7eca admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5ae53fa7d26844618db6b1ae8c887c89'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:43.094108 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-a5f4dd57-d445-4609-89c2-921be85e7eca admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5ae53fa7d26844618db6b1ae8c887c89'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:43.101993 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:55:43.105359 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-618764b3-dd83-4992-8c66-6a25fd022769 None None] MFA Rules not processed for user `dd25ccae1bcc4099bddee8a1259317a5`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:55:43.108443 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-618764b3-dd83-4992-8c66-6a25fd022769 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:43.134816 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 946/1874] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 00:55:42 2022] DELETE /identity/v3/users/5ae53fa7d26844618db6b1ae8c887c89 => generated 0 bytes in 303 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:55:43.139228 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3df07b5-e492-4a14-860c-50d0f3364e4e None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:43.139523 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3df07b5-e492-4a14-860c-50d0f3364e4e None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:43.140411 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3df07b5-e492-4a14-860c-50d0f3364e4e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:43.162759 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-f3df07b5-e492-4a14-860c-50d0f3364e4e None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:55:43.163214 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-f3df07b5-e492-4a14-860c-50d0f3364e4e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:43.226832 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 929/1875] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:55:43 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:55:43.242047 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-618764b3-dd83-4992-8c66-6a25fd022769 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:43.244624 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-618764b3-dd83-4992-8c66-6a25fd022769 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:43.247602 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 947/1876] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:55:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:55:43.258610 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3df07b5-e492-4a14-860c-50d0f3364e4e None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:43.260554 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-f3df07b5-e492-4a14-860c-50d0f3364e4e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:43.267782 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-618764b3-dd83-4992-8c66-6a25fd022769 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:43.332152 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9b28292-ab60-4f77-b8c6-7b0a9a195904 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:43.332794 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b28292-ab60-4f77-b8c6-7b0a9a195904 service cinder] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:43.333169 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b28292-ab60-4f77-b8c6-7b0a9a195904 service cinder] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:43.333526 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b28292-ab60-4f77-b8c6-7b0a9a195904 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:43.334534 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3df07b5-e492-4a14-860c-50d0f3364e4e None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:43.336625 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-e9b28292-ab60-4f77-b8c6-7b0a9a195904 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:43.338860 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2cdfbd4-d44f-4c55-b92c-90f98f33704a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:43.339494 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2cdfbd4-d44f-4c55-b92c-90f98f33704a service neutron] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:43.339899 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2cdfbd4-d44f-4c55-b92c-90f98f33704a service neutron] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:43.340191 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2cdfbd4-d44f-4c55-b92c-90f98f33704a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:43.349401 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:55:43.352004 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2cdfbd4-d44f-4c55-b92c-90f98f33704a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:43.353783 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2cdfbd4-d44f-4c55-b92c-90f98f33704a service neutron] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:43.366899 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 948/1877] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:55:43 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 28 00:55:43.449422 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9b28292-ab60-4f77-b8c6-7b0a9a195904 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:43.451196 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9b28292-ab60-4f77-b8c6-7b0a9a195904 service cinder] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:43.468111 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 930/1878] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:55:43 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 228 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:55:43.737202 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2cdfbd4-d44f-4c55-b92c-90f98f33704a service neutron] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:43.738820 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-d2cdfbd4-d44f-4c55-b92c-90f98f33704a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:43.755321 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2cdfbd4-d44f-4c55-b92c-90f98f33704a service neutron] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:43.762603 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9150964-9558-4bc3-a549-1d14d2d95772 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:43.763256 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9150964-9558-4bc3-a549-1d14d2d95772 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:43.763611 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9150964-9558-4bc3-a549-1d14d2d95772 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:43.763922 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9150964-9558-4bc3-a549-1d14d2d95772 admin admin] PATH_INFO: `/v3/projects/b56e9e0d8dd94c168bca305994863399` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:43.765122 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9150964-9558-4bc3-a549-1d14d2d95772 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b56e9e0d8dd94c168bca305994863399)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:43.767375 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9150964-9558-4bc3-a549-1d14d2d95772 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:43.876735 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-f9150964-9558-4bc3-a549-1d14d2d95772 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b56e9e0d8dd94c168bca305994863399', 'request_id': 'req-f9150964-9558-4bc3-a549-1d14d2d95772'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:43.888746 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-f9150964-9558-4bc3-a549-1d14d2d95772 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b56e9e0d8dd94c168bca305994863399', 'request_id': 'req-f9150964-9558-4bc3-a549-1d14d2d95772'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:43.909852 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-f9150964-9558-4bc3-a549-1d14d2d95772 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b56e9e0d8dd94c168bca305994863399', 'request_id': 'req-f9150964-9558-4bc3-a549-1d14d2d95772'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:43.927808 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-f9150964-9558-4bc3-a549-1d14d2d95772 admin admin] The token cache is being invalidate because project b56e9e0d8dd94c168bca305994863399 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81508) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 00:55:43.928108 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-f9150964-9558-4bc3-a549-1d14d2d95772 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:43.933196 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-f9150964-9558-4bc3-a549-1d14d2d95772 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:43.934822 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 949/1879] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 00:55:43 2022] DELETE /identity/v3/projects/b56e9e0d8dd94c168bca305994863399 => generated 0 bytes in 198 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:55:43.938972 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9b28292-ab60-4f77-b8c6-7b0a9a195904 service cinder] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:43.951121 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9b28292-ab60-4f77-b8c6-7b0a9a195904 service cinder] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:44.005328 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:55:44.015043 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac6c5bd5-7eac-4fec-a35f-230f7c7dd047 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:44.015764 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac6c5bd5-7eac-4fec-a35f-230f7c7dd047 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:44.016043 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac6c5bd5-7eac-4fec-a35f-230f7c7dd047 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:44.016490 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac6c5bd5-7eac-4fec-a35f-230f7c7dd047 admin admin] PATH_INFO: `/v3/users/97b6ba1a84814d48ae4cff2fd262a854` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:44.017678 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac6c5bd5-7eac-4fec-a35f-230f7c7dd047 admin admin] RBAC: Authorizing `identity:delete_user(user_id=97b6ba1a84814d48ae4cff2fd262a854)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:44.019647 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac6c5bd5-7eac-4fec-a35f-230f7c7dd047 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:44.077047 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-ac6c5bd5-7eac-4fec-a35f-230f7c7dd047 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '97b6ba1a84814d48ae4cff2fd262a854'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:44.091303 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-ac6c5bd5-7eac-4fec-a35f-230f7c7dd047 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '97b6ba1a84814d48ae4cff2fd262a854'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:44.101790 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-ac6c5bd5-7eac-4fec-a35f-230f7c7dd047 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '97b6ba1a84814d48ae4cff2fd262a854'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:44.125747 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-ac6c5bd5-7eac-4fec-a35f-230f7c7dd047 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '97b6ba1a84814d48ae4cff2fd262a854'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:44.126512 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-ac6c5bd5-7eac-4fec-a35f-230f7c7dd047 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '97b6ba1a84814d48ae4cff2fd262a854'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:44.136092 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 931/1880] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 00:55:43 2022] DELETE /identity/v3/users/97b6ba1a84814d48ae4cff2fd262a854 => generated 0 bytes in 198 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:55:44.672150 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9150964-9558-4bc3-a549-1d14d2d95772 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:44.673980 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-f9150964-9558-4bc3-a549-1d14d2d95772 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:44.712667 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9150964-9558-4bc3-a549-1d14d2d95772 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:44.867184 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9d7129b-3c63-4445-a2c1-e12d5ddb24fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:44.870094 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9d7129b-3c63-4445-a2c1-e12d5ddb24fd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:44.870739 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9d7129b-3c63-4445-a2c1-e12d5ddb24fd admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:44.871482 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9d7129b-3c63-4445-a2c1-e12d5ddb24fd admin admin] PATH_INFO: `/v3/projects/9293a9e25d5a4d43b4c3520f34e96d43` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:44.874027 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9d7129b-3c63-4445-a2c1-e12d5ddb24fd admin admin] RBAC: Authorizing `identity:delete_project(project_id=9293a9e25d5a4d43b4c3520f34e96d43)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:44.876985 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9d7129b-3c63-4445-a2c1-e12d5ddb24fd admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:44.977058 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-c9d7129b-3c63-4445-a2c1-e12d5ddb24fd admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9293a9e25d5a4d43b4c3520f34e96d43', 'request_id': 'req-c9d7129b-3c63-4445-a2c1-e12d5ddb24fd'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:44.996419 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-c9d7129b-3c63-4445-a2c1-e12d5ddb24fd admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9293a9e25d5a4d43b4c3520f34e96d43', 'request_id': 'req-c9d7129b-3c63-4445-a2c1-e12d5ddb24fd'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:45.007427 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-c9d7129b-3c63-4445-a2c1-e12d5ddb24fd admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9293a9e25d5a4d43b4c3520f34e96d43', 'request_id': 'req-c9d7129b-3c63-4445-a2c1-e12d5ddb24fd'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:45.048636 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-c9d7129b-3c63-4445-a2c1-e12d5ddb24fd admin admin] The token cache is being invalidate because project 9293a9e25d5a4d43b4c3520f34e96d43 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81508) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 00:55:45.048934 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-c9d7129b-3c63-4445-a2c1-e12d5ddb24fd admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:45.049626 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-c9d7129b-3c63-4445-a2c1-e12d5ddb24fd admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:45.055108 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 950/1881] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 00:55:44 2022] DELETE /identity/v3/projects/9293a9e25d5a4d43b4c3520f34e96d43 => generated 0 bytes in 383 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:55:45.060317 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24384383-7df2-4d64-8b96-960ddda69502 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:45.062139 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24384383-7df2-4d64-8b96-960ddda69502 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:45.062645 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24384383-7df2-4d64-8b96-960ddda69502 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:45.145894 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-24384383-7df2-4d64-8b96-960ddda69502 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:55:45.147724 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-24384383-7df2-4d64-8b96-960ddda69502 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:45.260292 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 932/1882] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:55:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:55:45.262070 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8a7defc-4643-4165-9915-5b9292c9f9c0 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:45.262070 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8a7defc-4643-4165-9915-5b9292c9f9c0 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:45.262269 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8a7defc-4643-4165-9915-5b9292c9f9c0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:45.289306 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-e8a7defc-4643-4165-9915-5b9292c9f9c0 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:55:45.291075 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-e8a7defc-4643-4165-9915-5b9292c9f9c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:45.308282 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 951/1883] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:55:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 45 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:55:45.309247 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24384383-7df2-4d64-8b96-960ddda69502 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:45.319344 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24384383-7df2-4d64-8b96-960ddda69502 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:45.324871 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd1c805e-b640-4baf-8234-a0513c49edfb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:45.325534 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd1c805e-b640-4baf-8234-a0513c49edfb admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:45.325810 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd1c805e-b640-4baf-8234-a0513c49edfb admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:45.326077 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd1c805e-b640-4baf-8234-a0513c49edfb admin admin] PATH_INFO: `/v3/domains` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:45.326731 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd1c805e-b640-4baf-8234-a0513c49edfb admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:55:45.327026 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd1c805e-b640-4baf-8234-a0513c49edfb admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:45.328826 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd1c805e-b640-4baf-8234-a0513c49edfb admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:45.345515 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8a7defc-4643-4165-9915-5b9292c9f9c0 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:45.347611 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-e8a7defc-4643-4165-9915-5b9292c9f9c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:45.353537 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:55:45.359374 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 933/1884] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Aug 28 00:55:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 28 00:55:45.362006 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd1c805e-b640-4baf-8234-a0513c49edfb admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:45.386724 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd1c805e-b640-4baf-8234-a0513c49edfb admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:45.395920 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5481eea7-ed77-4d82-86b8-8f1335885fb5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:45.399055 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5481eea7-ed77-4d82-86b8-8f1335885fb5 admin admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:45.399919 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5481eea7-ed77-4d82-86b8-8f1335885fb5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:45.400652 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5481eea7-ed77-4d82-86b8-8f1335885fb5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:45.402755 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5481eea7-ed77-4d82-86b8-8f1335885fb5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:45.406912 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5481eea7-ed77-4d82-86b8-8f1335885fb5 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:45.429455 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8a7defc-4643-4165-9915-5b9292c9f9c0 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:45.434952 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a77c661-ad8b-40a0-b717-9072ea128dd4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:45.436072 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a77c661-ad8b-40a0-b717-9072ea128dd4 service cinder] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:45.437914 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a77c661-ad8b-40a0-b717-9072ea128dd4 service cinder] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:45.438234 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a77c661-ad8b-40a0-b717-9072ea128dd4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:45.441995 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-7a77c661-ad8b-40a0-b717-9072ea128dd4 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:45.464728 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 934/1885] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 00:55:45 2022] POST /identity/v3/projects => generated 371 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 28 00:55:45.468917 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5481eea7-ed77-4d82-86b8-8f1335885fb5 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:45.540945 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a77c661-ad8b-40a0-b717-9072ea128dd4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:45.542498 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a77c661-ad8b-40a0-b717-9072ea128dd4 service cinder] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:45.562265 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 952/1886] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:55:45 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 217 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:55:45.567717 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:55:45.622004 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5481eea7-ed77-4d82-86b8-8f1335885fb5 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:45.627867 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e32a462-9e14-4d47-bf15-7497d075f382 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:45.628869 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e32a462-9e14-4d47-bf15-7497d075f382 admin admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:45.629353 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e32a462-9e14-4d47-bf15-7497d075f382 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:45.629777 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e32a462-9e14-4d47-bf15-7497d075f382 admin admin] PATH_INFO: `/v3/users` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:45.630569 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e32a462-9e14-4d47-bf15-7497d075f382 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:45.632740 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e32a462-9e14-4d47-bf15-7497d075f382 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:45.658538 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 935/1887] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 00:55:45 2022] POST /identity/v3/users => generated 286 bytes in 190 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 28 00:55:45.665022 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a77c661-ad8b-40a0-b717-9072ea128dd4 service cinder] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:45.680245 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a77c661-ad8b-40a0-b717-9072ea128dd4 service cinder] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:45.686524 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30a9d240-d3df-44da-9d87-fe14095930c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:45.687269 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30a9d240-d3df-44da-9d87-fe14095930c1 admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:45.688146 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30a9d240-d3df-44da-9d87-fe14095930c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:45.688449 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30a9d240-d3df-44da-9d87-fe14095930c1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:45.689134 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30a9d240-d3df-44da-9d87-fe14095930c1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:45.691102 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30a9d240-d3df-44da-9d87-fe14095930c1 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:45.705476 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 953/1888] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:55:45 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 00:55:45.711245 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e32a462-9e14-4d47-bf15-7497d075f382 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:45.727185 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e32a462-9e14-4d47-bf15-7497d075f382 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:45.735064 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b58cc159-1b14-431c-b9f5-c2462fbe4b10 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:45.735760 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b58cc159-1b14-431c-b9f5-c2462fbe4b10 admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:45.736039 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b58cc159-1b14-431c-b9f5-c2462fbe4b10 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:45.736338 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b58cc159-1b14-431c-b9f5-c2462fbe4b10 admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:45.737011 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b58cc159-1b14-431c-b9f5-c2462fbe4b10 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:45.738741 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b58cc159-1b14-431c-b9f5-c2462fbe4b10 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:45.751763 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 936/1889] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:55:45 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 00:55:45.759832 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30a9d240-d3df-44da-9d87-fe14095930c1 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:45.775069 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30a9d240-d3df-44da-9d87-fe14095930c1 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:45.781243 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41716304-9a84-40a3-b4c4-b453f7acc340 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:45.782153 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41716304-9a84-40a3-b4c4-b453f7acc340 admin admin] REQUEST_METHOD: `PUT` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:45.782629 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41716304-9a84-40a3-b4c4-b453f7acc340 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:45.783114 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41716304-9a84-40a3-b4c4-b453f7acc340 admin admin] PATH_INFO: `/v3/projects/14ca50fc28ee456aa9541287139ef44a/users/e7f487727f8b4d13a84fcb20a68b6af1/roles/0487ef7e943e4701915cb8ce0f2899d7` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:45.820768 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41716304-9a84-40a3-b4c4-b453f7acc340 admin admin] RBAC: Authorizing `identity:create_grant(project_id=14ca50fc28ee456aa9541287139ef44a, user_id=e7f487727f8b4d13a84fcb20a68b6af1, role_id=0487ef7e943e4701915cb8ce0f2899d7)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:45.826132 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41716304-9a84-40a3-b4c4-b453f7acc340 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:45.842930 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 954/1890] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 00:55:45 2022] PUT /identity/v3/projects/14ca50fc28ee456aa9541287139ef44a/users/e7f487727f8b4d13a84fcb20a68b6af1/roles/0487ef7e943e4701915cb8ce0f2899d7 => generated 0 bytes in 84 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:55:45.853554 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79e4e8e4-0e0e-405c-98e4-b8600c6e5675 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:45.854531 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79e4e8e4-0e0e-405c-98e4-b8600c6e5675 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:45.854815 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79e4e8e4-0e0e-405c-98e4-b8600c6e5675 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:45.889913 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-79e4e8e4-0e0e-405c-98e4-b8600c6e5675 None None] MFA Rules not processed for user `e7f487727f8b4d13a84fcb20a68b6af1`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:55:45.891569 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-79e4e8e4-0e0e-405c-98e4-b8600c6e5675 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:45.984915 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 937/1891] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:55:45 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 28 00:55:45.990499 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41716304-9a84-40a3-b4c4-b453f7acc340 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:46.056162 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41716304-9a84-40a3-b4c4-b453f7acc340 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:46.060920 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6dc2a2f-e93e-49be-9a12-eec30dcc360b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:46.061504 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6dc2a2f-e93e-49be-9a12-eec30dcc360b admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:46.061781 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6dc2a2f-e93e-49be-9a12-eec30dcc360b admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:46.062087 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6dc2a2f-e93e-49be-9a12-eec30dcc360b admin admin] PATH_INFO: `/v3/projects` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:46.062819 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6dc2a2f-e93e-49be-9a12-eec30dcc360b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:46.064744 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6dc2a2f-e93e-49be-9a12-eec30dcc360b admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:46.095344 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 955/1892] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 00:55:45 2022] POST /identity/v3/projects => generated 373 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:55:46.099548 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79e4e8e4-0e0e-405c-98e4-b8600c6e5675 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:46.110991 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79e4e8e4-0e0e-405c-98e4-b8600c6e5675 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:46.169988 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c919ae54-5f74-40de-99e4-a315f2894dad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:46.170609 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c919ae54-5f74-40de-99e4-a315f2894dad admin admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:46.170878 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c919ae54-5f74-40de-99e4-a315f2894dad admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:46.171185 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c919ae54-5f74-40de-99e4-a315f2894dad admin admin] PATH_INFO: `/v3/users` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:46.171814 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c919ae54-5f74-40de-99e4-a315f2894dad admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:46.173792 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c919ae54-5f74-40de-99e4-a315f2894dad admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:46.196546 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 938/1893] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 00:55:46 2022] POST /identity/v3/users => generated 293 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 28 00:55:46.201579 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6dc2a2f-e93e-49be-9a12-eec30dcc360b admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:46.212769 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6dc2a2f-e93e-49be-9a12-eec30dcc360b admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:46.218161 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01db1e47-70ed-44ba-aaac-a7e8cad5b008 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:46.218799 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01db1e47-70ed-44ba-aaac-a7e8cad5b008 admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:46.219076 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01db1e47-70ed-44ba-aaac-a7e8cad5b008 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:46.219353 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01db1e47-70ed-44ba-aaac-a7e8cad5b008 admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:46.219985 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01db1e47-70ed-44ba-aaac-a7e8cad5b008 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:46.221632 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01db1e47-70ed-44ba-aaac-a7e8cad5b008 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:46.223033 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:55:46.233810 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 956/1894] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:55:46 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 00:55:46.238428 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c919ae54-5f74-40de-99e4-a315f2894dad admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:46.248735 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c919ae54-5f74-40de-99e4-a315f2894dad admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:46.253541 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba9193b6-4efe-47b1-9bbd-ebe3a7e9e6d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:46.254217 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba9193b6-4efe-47b1-9bbd-ebe3a7e9e6d9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:46.254476 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba9193b6-4efe-47b1-9bbd-ebe3a7e9e6d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:46.254747 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba9193b6-4efe-47b1-9bbd-ebe3a7e9e6d9 admin admin] PATH_INFO: `/v3/projects/c4446703f72449a0af10d9e6b9579b0c/users/c25c026fe8db484f8212a12cde0b013a/roles/f08d6043261747e3b4763c744eef7dfb` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:46.271406 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba9193b6-4efe-47b1-9bbd-ebe3a7e9e6d9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c4446703f72449a0af10d9e6b9579b0c, user_id=c25c026fe8db484f8212a12cde0b013a, role_id=f08d6043261747e3b4763c744eef7dfb)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:46.274129 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba9193b6-4efe-47b1-9bbd-ebe3a7e9e6d9 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:46.289771 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 939/1895] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 00:55:46 2022] PUT /identity/v3/projects/c4446703f72449a0af10d9e6b9579b0c/users/c25c026fe8db484f8212a12cde0b013a/roles/f08d6043261747e3b4763c744eef7dfb => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:55:46.293860 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01db1e47-70ed-44ba-aaac-a7e8cad5b008 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:46.303975 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01db1e47-70ed-44ba-aaac-a7e8cad5b008 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:46.371471 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f16fb9c6-d476-4df4-9ec5-fb2fa5c390f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:46.371983 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f16fb9c6-d476-4df4-9ec5-fb2fa5c390f1 admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:46.372269 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f16fb9c6-d476-4df4-9ec5-fb2fa5c390f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:46.372535 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f16fb9c6-d476-4df4-9ec5-fb2fa5c390f1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:46.373203 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f16fb9c6-d476-4df4-9ec5-fb2fa5c390f1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:46.374868 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f16fb9c6-d476-4df4-9ec5-fb2fa5c390f1 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:46.385540 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 957/1896] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:55:46 2022] GET /identity/v3/roles => generated 1420 bytes in 92 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 00:55:46.389899 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba9193b6-4efe-47b1-9bbd-ebe3a7e9e6d9 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:46.400513 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba9193b6-4efe-47b1-9bbd-ebe3a7e9e6d9 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:46.405540 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-317f11e8-0e7a-4195-b082-9e85304d0b6a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:46.406240 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-317f11e8-0e7a-4195-b082-9e85304d0b6a admin admin] REQUEST_METHOD: `PUT` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:46.406525 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-317f11e8-0e7a-4195-b082-9e85304d0b6a admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:46.406800 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-317f11e8-0e7a-4195-b082-9e85304d0b6a admin admin] PATH_INFO: `/v3/projects/c4446703f72449a0af10d9e6b9579b0c/users/c25c026fe8db484f8212a12cde0b013a/roles/f08d6043261747e3b4763c744eef7dfb` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:46.408774 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-317f11e8-0e7a-4195-b082-9e85304d0b6a admin admin] RBAC: Authorizing `identity:create_grant(project_id=c4446703f72449a0af10d9e6b9579b0c, user_id=c25c026fe8db484f8212a12cde0b013a, role_id=f08d6043261747e3b4763c744eef7dfb)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:46.411431 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-317f11e8-0e7a-4195-b082-9e85304d0b6a admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:46.417501 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:55:46.423630 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 940/1897] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 00:55:46 2022] PUT /identity/v3/projects/c4446703f72449a0af10d9e6b9579b0c/users/c25c026fe8db484f8212a12cde0b013a/roles/f08d6043261747e3b4763c744eef7dfb => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:55:46.435113 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caefda17-fa5b-4b97-aaac-cb8477846c8d None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:46.435428 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caefda17-fa5b-4b97-aaac-cb8477846c8d None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:46.435690 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caefda17-fa5b-4b97-aaac-cb8477846c8d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:46.465255 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-caefda17-fa5b-4b97-aaac-cb8477846c8d None None] MFA Rules not processed for user `c25c026fe8db484f8212a12cde0b013a`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:55:46.466508 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-caefda17-fa5b-4b97-aaac-cb8477846c8d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:46.537957 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 958/1898] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:55:46 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 28 00:55:46.550978 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-317f11e8-0e7a-4195-b082-9e85304d0b6a admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:46.552698 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-317f11e8-0e7a-4195-b082-9e85304d0b6a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:46.645690 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-317f11e8-0e7a-4195-b082-9e85304d0b6a admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:46.649822 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11464e9a-0c74-4202-b756-268ae9eb079f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:46.650443 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11464e9a-0c74-4202-b756-268ae9eb079f service cinder] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:46.650718 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11464e9a-0c74-4202-b756-268ae9eb079f service cinder] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:46.650991 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11464e9a-0c74-4202-b756-268ae9eb079f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:46.663058 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11464e9a-0c74-4202-b756-268ae9eb079f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:46.665407 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11464e9a-0c74-4202-b756-268ae9eb079f service cinder] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:46.676395 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 941/1899] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:55:46 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 126 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:55:47.144010 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caefda17-fa5b-4b97-aaac-cb8477846c8d None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:47.221048 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caefda17-fa5b-4b97-aaac-cb8477846c8d None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:47.226516 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d6d7a44-fcd3-4524-9740-64c90cbbdf88 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:47.227164 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d6d7a44-fcd3-4524-9740-64c90cbbdf88 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:47.227442 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d6d7a44-fcd3-4524-9740-64c90cbbdf88 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:47.227711 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d6d7a44-fcd3-4524-9740-64c90cbbdf88 admin admin] PATH_INFO: `/v3/users/e7f487727f8b4d13a84fcb20a68b6af1` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:47.228937 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d6d7a44-fcd3-4524-9740-64c90cbbdf88 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e7f487727f8b4d13a84fcb20a68b6af1)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:47.230915 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d6d7a44-fcd3-4524-9740-64c90cbbdf88 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:47.255689 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=98,keystone:DELETE=2 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:55:47.284152 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-1d6d7a44-fcd3-4524-9740-64c90cbbdf88 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e7f487727f8b4d13a84fcb20a68b6af1'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:47.284792 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-1d6d7a44-fcd3-4524-9740-64c90cbbdf88 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e7f487727f8b4d13a84fcb20a68b6af1'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:47.296139 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-1d6d7a44-fcd3-4524-9740-64c90cbbdf88 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e7f487727f8b4d13a84fcb20a68b6af1'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:47.303609 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-1d6d7a44-fcd3-4524-9740-64c90cbbdf88 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e7f487727f8b4d13a84fcb20a68b6af1'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:47.304230 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-1d6d7a44-fcd3-4524-9740-64c90cbbdf88 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e7f487727f8b4d13a84fcb20a68b6af1'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:47.321245 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 959/1900] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 00:55:47 2022] DELETE /identity/v3/users/e7f487727f8b4d13a84fcb20a68b6af1 => generated 0 bytes in 178 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:55:47.326209 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3874c3c8-3d21-44ab-9b08-30be6499aa84 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:47.326724 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3874c3c8-3d21-44ab-9b08-30be6499aa84 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:47.326724 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3874c3c8-3d21-44ab-9b08-30be6499aa84 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:47.346206 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-3874c3c8-3d21-44ab-9b08-30be6499aa84 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:55:47.347408 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-3874c3c8-3d21-44ab-9b08-30be6499aa84 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:47.416214 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 942/1901] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:55:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:55:47.426160 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d6d7a44-fcd3-4524-9740-64c90cbbdf88 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:47.427628 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-1d6d7a44-fcd3-4524-9740-64c90cbbdf88 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:47.490185 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d6d7a44-fcd3-4524-9740-64c90cbbdf88 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:47.495736 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd4f535f-2010-4570-ba4f-d7346fd944e9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:47.496757 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd4f535f-2010-4570-ba4f-d7346fd944e9 service neutron] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:47.497165 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd4f535f-2010-4570-ba4f-d7346fd944e9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:47.497562 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd4f535f-2010-4570-ba4f-d7346fd944e9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:47.508380 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd4f535f-2010-4570-ba4f-d7346fd944e9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:47.509957 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd4f535f-2010-4570-ba4f-d7346fd944e9 service neutron] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:47.521079 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 960/1902] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:55:47 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 28 00:55:47.773173 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3874c3c8-3d21-44ab-9b08-30be6499aa84 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:47.774776 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-3874c3c8-3d21-44ab-9b08-30be6499aa84 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:47.814791 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd4f535f-2010-4570-ba4f-d7346fd944e9 service neutron] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:47.816352 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-cd4f535f-2010-4570-ba4f-d7346fd944e9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:47.830614 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd4f535f-2010-4570-ba4f-d7346fd944e9 service neutron] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:47.835242 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ade3fd15-d614-4b3a-af5d-66e40b14a401 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:47.835876 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ade3fd15-d614-4b3a-af5d-66e40b14a401 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:47.836153 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ade3fd15-d614-4b3a-af5d-66e40b14a401 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:47.836423 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ade3fd15-d614-4b3a-af5d-66e40b14a401 admin admin] PATH_INFO: `/v3/projects/14ca50fc28ee456aa9541287139ef44a` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:47.837568 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ade3fd15-d614-4b3a-af5d-66e40b14a401 admin admin] RBAC: Authorizing `identity:delete_project(project_id=14ca50fc28ee456aa9541287139ef44a)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:47.839477 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ade3fd15-d614-4b3a-af5d-66e40b14a401 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:47.841379 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3874c3c8-3d21-44ab-9b08-30be6499aa84 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:47.845114 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37535d86-f6eb-42d4-ba0d-7c91165fd976 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:47.845734 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37535d86-f6eb-42d4-ba0d-7c91165fd976 service cinder] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:47.846148 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37535d86-f6eb-42d4-ba0d-7c91165fd976 service cinder] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:47.846502 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37535d86-f6eb-42d4-ba0d-7c91165fd976 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:47.848515 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-37535d86-f6eb-42d4-ba0d-7c91165fd976 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:47.901872 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-ade3fd15-d614-4b3a-af5d-66e40b14a401 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '14ca50fc28ee456aa9541287139ef44a', 'request_id': 'req-ade3fd15-d614-4b3a-af5d-66e40b14a401'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:47.907707 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:55:47.908384 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-ade3fd15-d614-4b3a-af5d-66e40b14a401 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '14ca50fc28ee456aa9541287139ef44a', 'request_id': 'req-ade3fd15-d614-4b3a-af5d-66e40b14a401'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:47.915342 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-ade3fd15-d614-4b3a-af5d-66e40b14a401 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '14ca50fc28ee456aa9541287139ef44a', 'request_id': 'req-ade3fd15-d614-4b3a-af5d-66e40b14a401'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:47.921089 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37535d86-f6eb-42d4-ba0d-7c91165fd976 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:47.922560 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37535d86-f6eb-42d4-ba0d-7c91165fd976 service cinder] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:47.933109 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 943/1903] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:55:47 2022] GET /identity/v3/auth/tokens => generated 3319 bytes in 161 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:55:47.937423 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-ade3fd15-d614-4b3a-af5d-66e40b14a401 admin admin] The token cache is being invalidate because project 14ca50fc28ee456aa9541287139ef44a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81508) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 00:55:47.939383 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-ade3fd15-d614-4b3a-af5d-66e40b14a401 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:47.940421 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-ade3fd15-d614-4b3a-af5d-66e40b14a401 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:47.943037 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 961/1904] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 00:55:47 2022] DELETE /identity/v3/projects/14ca50fc28ee456aa9541287139ef44a => generated 0 bytes in 129 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 28 00:55:47.947191 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37535d86-f6eb-42d4-ba0d-7c91165fd976 service cinder] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:47.958768 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37535d86-f6eb-42d4-ba0d-7c91165fd976 service cinder] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:48.039678 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ceb8069-2de1-409e-b247-3137111396a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:48.040331 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ceb8069-2de1-409e-b247-3137111396a8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:48.040613 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ceb8069-2de1-409e-b247-3137111396a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:48.040881 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ceb8069-2de1-409e-b247-3137111396a8 admin admin] PATH_INFO: `/v3/users/c25c026fe8db484f8212a12cde0b013a` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:48.042213 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ceb8069-2de1-409e-b247-3137111396a8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c25c026fe8db484f8212a12cde0b013a)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:48.044138 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ceb8069-2de1-409e-b247-3137111396a8 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:48.099541 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-9ceb8069-2de1-409e-b247-3137111396a8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c25c026fe8db484f8212a12cde0b013a'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:48.117262 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-9ceb8069-2de1-409e-b247-3137111396a8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c25c026fe8db484f8212a12cde0b013a'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:48.133967 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-9ceb8069-2de1-409e-b247-3137111396a8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c25c026fe8db484f8212a12cde0b013a'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:48.144653 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-9ceb8069-2de1-409e-b247-3137111396a8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c25c026fe8db484f8212a12cde0b013a'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:48.145657 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-9ceb8069-2de1-409e-b247-3137111396a8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c25c026fe8db484f8212a12cde0b013a'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:48.152526 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 944/1905] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 00:55:47 2022] DELETE /identity/v3/users/c25c026fe8db484f8212a12cde0b013a => generated 0 bytes in 206 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:55:48.603958 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ade3fd15-d614-4b3a-af5d-66e40b14a401 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:48.606021 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-ade3fd15-d614-4b3a-af5d-66e40b14a401 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:48.621788 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ade3fd15-d614-4b3a-af5d-66e40b14a401 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:48.641258 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:DELETE=15,keystone:SELECT=83,keystone:INSERT=2 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:55:48.680158 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebdd6b9e-bcd3-4102-809f-082052cdd198 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:48.680788 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebdd6b9e-bcd3-4102-809f-082052cdd198 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:48.681093 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebdd6b9e-bcd3-4102-809f-082052cdd198 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:48.681402 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebdd6b9e-bcd3-4102-809f-082052cdd198 admin admin] PATH_INFO: `/v3/projects/c4446703f72449a0af10d9e6b9579b0c` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:48.682516 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebdd6b9e-bcd3-4102-809f-082052cdd198 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c4446703f72449a0af10d9e6b9579b0c)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:48.684417 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebdd6b9e-bcd3-4102-809f-082052cdd198 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:48.735305 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-ebdd6b9e-bcd3-4102-809f-082052cdd198 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c4446703f72449a0af10d9e6b9579b0c', 'request_id': 'req-ebdd6b9e-bcd3-4102-809f-082052cdd198'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:48.741142 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-ebdd6b9e-bcd3-4102-809f-082052cdd198 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c4446703f72449a0af10d9e6b9579b0c', 'request_id': 'req-ebdd6b9e-bcd3-4102-809f-082052cdd198'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:48.747676 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-ebdd6b9e-bcd3-4102-809f-082052cdd198 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c4446703f72449a0af10d9e6b9579b0c', 'request_id': 'req-ebdd6b9e-bcd3-4102-809f-082052cdd198'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:48.764107 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-ebdd6b9e-bcd3-4102-809f-082052cdd198 admin admin] The token cache is being invalidate because project c4446703f72449a0af10d9e6b9579b0c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81508) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 00:55:48.764391 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-ebdd6b9e-bcd3-4102-809f-082052cdd198 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:48.765133 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-ebdd6b9e-bcd3-4102-809f-082052cdd198 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:48.766708 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 962/1906] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 00:55:48 2022] DELETE /identity/v3/projects/c4446703f72449a0af10d9e6b9579b0c => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:55:48.771921 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5b168df-4ed9-4599-9023-634ecfa04dfc None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:48.772407 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5b168df-4ed9-4599-9023-634ecfa04dfc None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:48.772499 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5b168df-4ed9-4599-9023-634ecfa04dfc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:48.793760 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-c5b168df-4ed9-4599-9023-634ecfa04dfc None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:55:48.795029 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-c5b168df-4ed9-4599-9023-634ecfa04dfc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:48.858117 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 945/1907] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:55:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 28 00:55:48.866144 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2be9963-07e4-4146-a6ee-ad47f1ffe3a0 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:48.866427 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2be9963-07e4-4146-a6ee-ad47f1ffe3a0 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:48.866702 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2be9963-07e4-4146-a6ee-ad47f1ffe3a0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:48.887153 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-a2be9963-07e4-4146-a6ee-ad47f1ffe3a0 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:55:48.888381 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-a2be9963-07e4-4146-a6ee-ad47f1ffe3a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:48.896659 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 963/1908] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:55:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:55:48.901209 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5b168df-4ed9-4599-9023-634ecfa04dfc None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:48.912150 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5b168df-4ed9-4599-9023-634ecfa04dfc None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:48.918037 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac0ada05-e72d-49cd-b3b7-22cb301cffda admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:48.918632 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac0ada05-e72d-49cd-b3b7-22cb301cffda admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:48.918913 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac0ada05-e72d-49cd-b3b7-22cb301cffda admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:48.919173 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac0ada05-e72d-49cd-b3b7-22cb301cffda admin admin] PATH_INFO: `/v3/domains` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:48.919830 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac0ada05-e72d-49cd-b3b7-22cb301cffda admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:55:48.920155 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac0ada05-e72d-49cd-b3b7-22cb301cffda admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:48.921805 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac0ada05-e72d-49cd-b3b7-22cb301cffda admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:48.928925 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:55:48.938639 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 946/1909] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Aug 28 00:55:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 28 00:55:48.939892 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2be9963-07e4-4146-a6ee-ad47f1ffe3a0 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:48.949895 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2be9963-07e4-4146-a6ee-ad47f1ffe3a0 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:48.954985 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f88008c0-4617-4ca1-bab9-be6d74ccfb12 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:48.955681 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f88008c0-4617-4ca1-bab9-be6d74ccfb12 admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:48.956401 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f88008c0-4617-4ca1-bab9-be6d74ccfb12 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:48.956401 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f88008c0-4617-4ca1-bab9-be6d74ccfb12 admin admin] PATH_INFO: `/v3/projects` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:48.957084 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f88008c0-4617-4ca1-bab9-be6d74ccfb12 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:48.960683 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f88008c0-4617-4ca1-bab9-be6d74ccfb12 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:48.999525 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 964/1910] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 00:55:48 2022] POST /identity/v3/projects => generated 365 bytes in 60 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:55:49.001969 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac0ada05-e72d-49cd-b3b7-22cb301cffda admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:49.016895 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac0ada05-e72d-49cd-b3b7-22cb301cffda admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:49.079686 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f65b019-4101-4dae-90db-72946b58fdee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:49.080374 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f65b019-4101-4dae-90db-72946b58fdee admin admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:49.080549 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f65b019-4101-4dae-90db-72946b58fdee admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:49.080818 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f65b019-4101-4dae-90db-72946b58fdee admin admin] PATH_INFO: `/v3/users` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:49.081491 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f65b019-4101-4dae-90db-72946b58fdee admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:49.083508 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f65b019-4101-4dae-90db-72946b58fdee admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:49.106091 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 947/1911] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 00:55:49 2022] POST /identity/v3/users => generated 283 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:55:49.110725 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f88008c0-4617-4ca1-bab9-be6d74ccfb12 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:49.128246 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f88008c0-4617-4ca1-bab9-be6d74ccfb12 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:49.134679 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b83f86da-f87e-4a9e-bd6c-2e57b1618c68 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:49.137018 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b83f86da-f87e-4a9e-bd6c-2e57b1618c68 admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:49.137306 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b83f86da-f87e-4a9e-bd6c-2e57b1618c68 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:49.137580 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b83f86da-f87e-4a9e-bd6c-2e57b1618c68 admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:49.138298 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b83f86da-f87e-4a9e-bd6c-2e57b1618c68 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:49.140005 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b83f86da-f87e-4a9e-bd6c-2e57b1618c68 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:49.152253 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 965/1912] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:55:49 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 00:55:49.156493 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f65b019-4101-4dae-90db-72946b58fdee admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:49.171077 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f65b019-4101-4dae-90db-72946b58fdee admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:49.176628 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-666e6916-f2fc-4caf-b1d5-355f74900a03 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:49.177371 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-666e6916-f2fc-4caf-b1d5-355f74900a03 admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:49.177723 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-666e6916-f2fc-4caf-b1d5-355f74900a03 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:49.178032 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-666e6916-f2fc-4caf-b1d5-355f74900a03 admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:49.178872 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-666e6916-f2fc-4caf-b1d5-355f74900a03 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:49.180682 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-666e6916-f2fc-4caf-b1d5-355f74900a03 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:49.197580 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 948/1913] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:55:49 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 00:55:49.208433 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b83f86da-f87e-4a9e-bd6c-2e57b1618c68 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:49.219083 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b83f86da-f87e-4a9e-bd6c-2e57b1618c68 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:49.224805 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26ffced0-bed9-40b5-8e69-0389e05c545b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:49.225527 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26ffced0-bed9-40b5-8e69-0389e05c545b admin admin] REQUEST_METHOD: `PUT` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:49.225819 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26ffced0-bed9-40b5-8e69-0389e05c545b admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:49.226072 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26ffced0-bed9-40b5-8e69-0389e05c545b admin admin] PATH_INFO: `/v3/projects/a26f0057529a455c9c5b3bfb363c4b4e/users/688deb3207f1443f8d52cd9ce250892e/roles/0487ef7e943e4701915cb8ce0f2899d7` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:49.245598 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26ffced0-bed9-40b5-8e69-0389e05c545b admin admin] RBAC: Authorizing `identity:create_grant(project_id=a26f0057529a455c9c5b3bfb363c4b4e, user_id=688deb3207f1443f8d52cd9ce250892e, role_id=0487ef7e943e4701915cb8ce0f2899d7)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:49.248594 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26ffced0-bed9-40b5-8e69-0389e05c545b admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:49.263796 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 966/1914] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 00:55:49 2022] PUT /identity/v3/projects/a26f0057529a455c9c5b3bfb363c4b4e/users/688deb3207f1443f8d52cd9ce250892e/roles/0487ef7e943e4701915cb8ce0f2899d7 => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:55:49.275093 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57490439-bc6f-47c7-801f-76bf2ff3a1c4 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:49.275380 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57490439-bc6f-47c7-801f-76bf2ff3a1c4 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:49.275652 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57490439-bc6f-47c7-801f-76bf2ff3a1c4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:49.304669 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-57490439-bc6f-47c7-801f-76bf2ff3a1c4 None None] MFA Rules not processed for user `688deb3207f1443f8d52cd9ce250892e`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:55:49.306012 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-57490439-bc6f-47c7-801f-76bf2ff3a1c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:49.384697 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 949/1915] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:55:49 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:55:49.389602 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26ffced0-bed9-40b5-8e69-0389e05c545b admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:49.441859 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:55:49.467600 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26ffced0-bed9-40b5-8e69-0389e05c545b admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:49.473222 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4c607b0-5b92-4429-88e4-bd2fed686bde admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:49.474042 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4c607b0-5b92-4429-88e4-bd2fed686bde admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:49.474337 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4c607b0-5b92-4429-88e4-bd2fed686bde admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:49.474684 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4c607b0-5b92-4429-88e4-bd2fed686bde admin admin] PATH_INFO: `/v3/projects` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:49.475516 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4c607b0-5b92-4429-88e4-bd2fed686bde admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:49.477559 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4c607b0-5b92-4429-88e4-bd2fed686bde admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:49.510148 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 967/1916] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 00:55:49 2022] POST /identity/v3/projects => generated 365 bytes in 121 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:55:49.514402 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57490439-bc6f-47c7-801f-76bf2ff3a1c4 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:49.525758 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57490439-bc6f-47c7-801f-76bf2ff3a1c4 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:49.595773 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20d18565-a2be-45a7-988a-7d63ef3aac86 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:49.596509 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20d18565-a2be-45a7-988a-7d63ef3aac86 admin admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:49.596844 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20d18565-a2be-45a7-988a-7d63ef3aac86 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:49.597199 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20d18565-a2be-45a7-988a-7d63ef3aac86 admin admin] PATH_INFO: `/v3/users` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:49.597888 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20d18565-a2be-45a7-988a-7d63ef3aac86 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:49.599957 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20d18565-a2be-45a7-988a-7d63ef3aac86 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:49.612307 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:55:49.634388 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 950/1917] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 00:55:49 2022] POST /identity/v3/users => generated 289 bytes in 118 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:55:49.635755 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4c607b0-5b92-4429-88e4-bd2fed686bde admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:49.646908 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4c607b0-5b92-4429-88e4-bd2fed686bde admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:49.653854 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16d5e77d-5bd3-44e9-a810-97aa3d7eea91 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:49.654731 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16d5e77d-5bd3-44e9-a810-97aa3d7eea91 admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:49.655009 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16d5e77d-5bd3-44e9-a810-97aa3d7eea91 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:49.655283 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16d5e77d-5bd3-44e9-a810-97aa3d7eea91 admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:49.656201 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16d5e77d-5bd3-44e9-a810-97aa3d7eea91 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:49.658671 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16d5e77d-5bd3-44e9-a810-97aa3d7eea91 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:49.682675 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 968/1918] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:55:49 2022] GET /identity/v3/roles => generated 1420 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 00:55:49.686821 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20d18565-a2be-45a7-988a-7d63ef3aac86 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:49.698020 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20d18565-a2be-45a7-988a-7d63ef3aac86 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:49.708993 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e727498-d856-4edd-b630-ea3d4b18a2e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:49.709761 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e727498-d856-4edd-b630-ea3d4b18a2e4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:49.710076 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e727498-d856-4edd-b630-ea3d4b18a2e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:49.710301 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e727498-d856-4edd-b630-ea3d4b18a2e4 admin admin] PATH_INFO: `/v3/projects/ec15f147ec06441d99263f79a527958e/users/31c6473ee616414e9aef96b474e13e45/roles/f08d6043261747e3b4763c744eef7dfb` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:49.731900 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e727498-d856-4edd-b630-ea3d4b18a2e4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ec15f147ec06441d99263f79a527958e, user_id=31c6473ee616414e9aef96b474e13e45, role_id=f08d6043261747e3b4763c744eef7dfb)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:49.734708 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e727498-d856-4edd-b630-ea3d4b18a2e4 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:49.748102 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 951/1919] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 00:55:49 2022] PUT /identity/v3/projects/ec15f147ec06441d99263f79a527958e/users/31c6473ee616414e9aef96b474e13e45/roles/f08d6043261747e3b4763c744eef7dfb => generated 0 bytes in 62 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:55:49.752205 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16d5e77d-5bd3-44e9-a810-97aa3d7eea91 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:49.762710 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16d5e77d-5bd3-44e9-a810-97aa3d7eea91 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:49.839546 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d1cba04-9ede-4d14-8655-f95778b466c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:49.840221 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d1cba04-9ede-4d14-8655-f95778b466c4 admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:49.840514 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d1cba04-9ede-4d14-8655-f95778b466c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:49.840784 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d1cba04-9ede-4d14-8655-f95778b466c4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:49.841461 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d1cba04-9ede-4d14-8655-f95778b466c4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:49.843619 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d1cba04-9ede-4d14-8655-f95778b466c4 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:49.870628 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 969/1920] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:55:49 2022] GET /identity/v3/roles => generated 1420 bytes in 119 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 28 00:55:49.875036 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e727498-d856-4edd-b630-ea3d4b18a2e4 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:49.890463 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e727498-d856-4edd-b630-ea3d4b18a2e4 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:49.895440 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-848cb9bf-fd7d-4cb0-838e-b06a19afb0d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:49.896225 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-848cb9bf-fd7d-4cb0-838e-b06a19afb0d4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:49.896578 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-848cb9bf-fd7d-4cb0-838e-b06a19afb0d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:49.896921 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-848cb9bf-fd7d-4cb0-838e-b06a19afb0d4 admin admin] PATH_INFO: `/v3/projects/ec15f147ec06441d99263f79a527958e/users/31c6473ee616414e9aef96b474e13e45/roles/f08d6043261747e3b4763c744eef7dfb` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:49.899077 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-848cb9bf-fd7d-4cb0-838e-b06a19afb0d4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ec15f147ec06441d99263f79a527958e, user_id=31c6473ee616414e9aef96b474e13e45, role_id=f08d6043261747e3b4763c744eef7dfb)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:49.902065 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-848cb9bf-fd7d-4cb0-838e-b06a19afb0d4 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:49.921122 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 952/1921] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 00:55:49 2022] PUT /identity/v3/projects/ec15f147ec06441d99263f79a527958e/users/31c6473ee616414e9aef96b474e13e45/roles/f08d6043261747e3b4763c744eef7dfb => generated 0 bytes in 43 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:55:49.928534 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-259a347f-0dac-4030-8614-18364997455e None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:49.928931 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-259a347f-0dac-4030-8614-18364997455e None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:49.929311 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-259a347f-0dac-4030-8614-18364997455e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:49.984476 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-259a347f-0dac-4030-8614-18364997455e None None] MFA Rules not processed for user `31c6473ee616414e9aef96b474e13e45`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:55:49.986041 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-259a347f-0dac-4030-8614-18364997455e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:50.077466 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 970/1922] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:55:49 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 146 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:55:50.085818 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-848cb9bf-fd7d-4cb0-838e-b06a19afb0d4 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:50.087512 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-848cb9bf-fd7d-4cb0-838e-b06a19afb0d4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:50.155823 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-848cb9bf-fd7d-4cb0-838e-b06a19afb0d4 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:50.159919 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaa166de-fecf-4edd-ad87-e1d8c7447d25 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:50.160663 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaa166de-fecf-4edd-ad87-e1d8c7447d25 service cinder] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:50.160816 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaa166de-fecf-4edd-ad87-e1d8c7447d25 service cinder] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:50.161369 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaa166de-fecf-4edd-ad87-e1d8c7447d25 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:50.171317 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaa166de-fecf-4edd-ad87-e1d8c7447d25 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:50.172861 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaa166de-fecf-4edd-ad87-e1d8c7447d25 service cinder] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:50.183015 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 953/1923] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:55:50 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:55:50.293021 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-259a347f-0dac-4030-8614-18364997455e None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:50.303517 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-259a347f-0dac-4030-8614-18364997455e None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:50.307937 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cae16d3-a314-4ee3-b108-bc8729067741 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:50.308598 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cae16d3-a314-4ee3-b108-bc8729067741 service cinder] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:50.308869 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cae16d3-a314-4ee3-b108-bc8729067741 service cinder] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:50.309318 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cae16d3-a314-4ee3-b108-bc8729067741 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:50.359637 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:55:50.378849 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cae16d3-a314-4ee3-b108-bc8729067741 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:50.380816 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cae16d3-a314-4ee3-b108-bc8729067741 service cinder] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:50.397147 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 971/1924] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:55:50 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:55:53.426594 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaa166de-fecf-4edd-ad87-e1d8c7447d25 service cinder] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:53.428478 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-aaa166de-fecf-4edd-ad87-e1d8c7447d25 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:53.500281 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cae16d3-a314-4ee3-b108-bc8729067741 service cinder] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:53.503358 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-9cae16d3-a314-4ee3-b108-bc8729067741 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:53.528828 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaa166de-fecf-4edd-ad87-e1d8c7447d25 service cinder] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:53.537924 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8568ee9a-1001-4ef2-b9f0-374124a08a0e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:53.538836 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8568ee9a-1001-4ef2-b9f0-374124a08a0e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:53.539289 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8568ee9a-1001-4ef2-b9f0-374124a08a0e admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:53.539733 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8568ee9a-1001-4ef2-b9f0-374124a08a0e admin admin] PATH_INFO: `/v3/users/8e0d0d02bcab491f98409142789d5212` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:53.541440 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8568ee9a-1001-4ef2-b9f0-374124a08a0e admin admin] RBAC: Authorizing `identity:delete_user(user_id=8e0d0d02bcab491f98409142789d5212)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:53.543555 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8568ee9a-1001-4ef2-b9f0-374124a08a0e admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:53.600258 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cae16d3-a314-4ee3-b108-bc8729067741 service cinder] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:53.606097 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-892d7fcd-21f4-48ed-a54f-8ef363714ed0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:53.607115 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-892d7fcd-21f4-48ed-a54f-8ef363714ed0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:53.607839 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-892d7fcd-21f4-48ed-a54f-8ef363714ed0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:53.608265 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-892d7fcd-21f4-48ed-a54f-8ef363714ed0 admin admin] PATH_INFO: `/v3/users/2c9ba24f2fc142aa95949aeb2a5551b0` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:53.609119 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-8568ee9a-1001-4ef2-b9f0-374124a08a0e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8e0d0d02bcab491f98409142789d5212'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:53.610241 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-892d7fcd-21f4-48ed-a54f-8ef363714ed0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2c9ba24f2fc142aa95949aeb2a5551b0)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:53.613229 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-892d7fcd-21f4-48ed-a54f-8ef363714ed0 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:53.626614 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=89,keystone:INSERT=2,keystone:DELETE=9 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:55:53.629110 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-8568ee9a-1001-4ef2-b9f0-374124a08a0e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8e0d0d02bcab491f98409142789d5212'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:53.648529 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-8568ee9a-1001-4ef2-b9f0-374124a08a0e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8e0d0d02bcab491f98409142789d5212'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:53.659693 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-8568ee9a-1001-4ef2-b9f0-374124a08a0e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8e0d0d02bcab491f98409142789d5212'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:53.660617 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-8568ee9a-1001-4ef2-b9f0-374124a08a0e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8e0d0d02bcab491f98409142789d5212'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:53.668620 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 954/1925] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 00:55:53 2022] DELETE /identity/v3/users/8e0d0d02bcab491f98409142789d5212 => generated 0 bytes in 242 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:55:53.669380 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8568ee9a-1001-4ef2-b9f0-374124a08a0e admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:53.672600 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-8568ee9a-1001-4ef2-b9f0-374124a08a0e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:53.685474 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-892d7fcd-21f4-48ed-a54f-8ef363714ed0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2c9ba24f2fc142aa95949aeb2a5551b0'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:53.686210 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-892d7fcd-21f4-48ed-a54f-8ef363714ed0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2c9ba24f2fc142aa95949aeb2a5551b0'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:53.697964 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-892d7fcd-21f4-48ed-a54f-8ef363714ed0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2c9ba24f2fc142aa95949aeb2a5551b0'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:53.706583 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-892d7fcd-21f4-48ed-a54f-8ef363714ed0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2c9ba24f2fc142aa95949aeb2a5551b0'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:53.707429 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-892d7fcd-21f4-48ed-a54f-8ef363714ed0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2c9ba24f2fc142aa95949aeb2a5551b0'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:53.729736 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 972/1926] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 00:55:53 2022] DELETE /identity/v3/users/2c9ba24f2fc142aa95949aeb2a5551b0 => generated 0 bytes in 230 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:55:53.730963 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9731a148-fdd7-422e-92b2-a042ad51a061 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:53.731341 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9731a148-fdd7-422e-92b2-a042ad51a061 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:53.731724 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9731a148-fdd7-422e-92b2-a042ad51a061 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:53.753044 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-9731a148-fdd7-422e-92b2-a042ad51a061 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:55:53.754256 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-9731a148-fdd7-422e-92b2-a042ad51a061 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:53.772273 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8568ee9a-1001-4ef2-b9f0-374124a08a0e admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:53.821285 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 973/1927] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:55:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:55:53.822652 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f2abc9-3240-4850-8a5b-f210587e08cf None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:53.822933 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f2abc9-3240-4850-8a5b-f210587e08cf None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:53.823202 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f2abc9-3240-4850-8a5b-f210587e08cf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:53.832003 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:55:53.846779 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-47f2abc9-3240-4850-8a5b-f210587e08cf None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:55:53.848113 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-47f2abc9-3240-4850-8a5b-f210587e08cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:53.849961 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e087f2dc-9acb-48bd-bf10-983d010e0fae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:53.850718 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e087f2dc-9acb-48bd-bf10-983d010e0fae admin admin] REQUEST_METHOD: `DELETE` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:53.851114 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e087f2dc-9acb-48bd-bf10-983d010e0fae admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:53.859119 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e087f2dc-9acb-48bd-bf10-983d010e0fae admin admin] PATH_INFO: `/v3/users/688deb3207f1443f8d52cd9ce250892e` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:53.859724 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e087f2dc-9acb-48bd-bf10-983d010e0fae admin admin] RBAC: Authorizing `identity:delete_user(user_id=688deb3207f1443f8d52cd9ce250892e)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:53.862229 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e087f2dc-9acb-48bd-bf10-983d010e0fae admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:53.865906 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 974/1928] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:55:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:55:53.868752 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47f2abc9-3240-4850-8a5b-f210587e08cf None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:53.871586 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-47f2abc9-3240-4850-8a5b-f210587e08cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:53.917320 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-e087f2dc-9acb-48bd-bf10-983d010e0fae admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '688deb3207f1443f8d52cd9ce250892e'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:53.931668 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-e087f2dc-9acb-48bd-bf10-983d010e0fae admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '688deb3207f1443f8d52cd9ce250892e'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:53.940506 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47f2abc9-3240-4850-8a5b-f210587e08cf None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:53.940915 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-e087f2dc-9acb-48bd-bf10-983d010e0fae admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '688deb3207f1443f8d52cd9ce250892e'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:53.944516 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4b7d94a-517b-4ef5-8de1-6da08be23ef7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:53.945248 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4b7d94a-517b-4ef5-8de1-6da08be23ef7 service neutron] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:53.945589 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4b7d94a-517b-4ef5-8de1-6da08be23ef7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:53.945915 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4b7d94a-517b-4ef5-8de1-6da08be23ef7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:53.949467 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-e087f2dc-9acb-48bd-bf10-983d010e0fae admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '688deb3207f1443f8d52cd9ce250892e'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:53.950494 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-e087f2dc-9acb-48bd-bf10-983d010e0fae admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '688deb3207f1443f8d52cd9ce250892e'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:53.956077 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4b7d94a-517b-4ef5-8de1-6da08be23ef7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:53.957800 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4b7d94a-517b-4ef5-8de1-6da08be23ef7 service neutron] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:53.962065 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 955/1929] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 00:55:53 2022] DELETE /identity/v3/users/688deb3207f1443f8d52cd9ce250892e => generated 0 bytes in 293 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:55:53.966335 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e087f2dc-9acb-48bd-bf10-983d010e0fae admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:53.968075 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-e087f2dc-9acb-48bd-bf10-983d010e0fae admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:53.970443 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 975/1930] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:55:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:55:53.971774 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98ae652c-06e0-42ee-be2e-b9a2290b3c38 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:53.972156 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98ae652c-06e0-42ee-be2e-b9a2290b3c38 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:53.972523 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98ae652c-06e0-42ee-be2e-b9a2290b3c38 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:53.990141 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=87,keystone:INSERT=2,keystone:DELETE=11 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:55:53.997051 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-98ae652c-06e0-42ee-be2e-b9a2290b3c38 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:55:54.000148 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-98ae652c-06e0-42ee-be2e-b9a2290b3c38 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:54.064406 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e087f2dc-9acb-48bd-bf10-983d010e0fae admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:54.070724 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98919e23-8c35-417f-9eac-9c131772d96a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:54.071347 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98919e23-8c35-417f-9eac-9c131772d96a service neutron] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:54.071700 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98919e23-8c35-417f-9eac-9c131772d96a service neutron] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:54.072360 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98919e23-8c35-417f-9eac-9c131772d96a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:54.075261 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-98919e23-8c35-417f-9eac-9c131772d96a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:54.091080 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 976/1931] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:55:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:55:54.103706 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98ae652c-06e0-42ee-be2e-b9a2290b3c38 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:54.125839 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98ae652c-06e0-42ee-be2e-b9a2290b3c38 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:54.137625 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4637e785-6497-4f98-98dd-87137e9e84fe service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:54.138266 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4637e785-6497-4f98-98dd-87137e9e84fe service neutron] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:54.138549 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4637e785-6497-4f98-98dd-87137e9e84fe service neutron] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:54.138825 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4637e785-6497-4f98-98dd-87137e9e84fe service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:54.166171 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4637e785-6497-4f98-98dd-87137e9e84fe service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:54.167665 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4637e785-6497-4f98-98dd-87137e9e84fe service neutron] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:54.185136 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 977/1932] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:55:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:55:54.190985 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98919e23-8c35-417f-9eac-9c131772d96a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:54.192449 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98919e23-8c35-417f-9eac-9c131772d96a service neutron] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:54.219330 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 956/1933] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:55:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 257 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:55:54.617055 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4637e785-6497-4f98-98dd-87137e9e84fe service neutron] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:54.622267 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-4637e785-6497-4f98-98dd-87137e9e84fe service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:54.643622 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4637e785-6497-4f98-98dd-87137e9e84fe service neutron] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:54.664874 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d016ac18-5a16-4836-882f-b2d3bce4e48a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:54.667397 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d016ac18-5a16-4836-882f-b2d3bce4e48a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:54.668509 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d016ac18-5a16-4836-882f-b2d3bce4e48a admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:54.670339 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d016ac18-5a16-4836-882f-b2d3bce4e48a admin admin] PATH_INFO: `/v3/projects/72d44c3f93034acfa2eeb77cb85a66b9` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:54.674473 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d016ac18-5a16-4836-882f-b2d3bce4e48a admin admin] RBAC: Authorizing `identity:delete_project(project_id=72d44c3f93034acfa2eeb77cb85a66b9)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:54.678159 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d016ac18-5a16-4836-882f-b2d3bce4e48a admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:54.759862 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-d016ac18-5a16-4836-882f-b2d3bce4e48a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '72d44c3f93034acfa2eeb77cb85a66b9', 'request_id': 'req-d016ac18-5a16-4836-882f-b2d3bce4e48a'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:54.766334 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-d016ac18-5a16-4836-882f-b2d3bce4e48a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '72d44c3f93034acfa2eeb77cb85a66b9', 'request_id': 'req-d016ac18-5a16-4836-882f-b2d3bce4e48a'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:54.774847 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-d016ac18-5a16-4836-882f-b2d3bce4e48a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '72d44c3f93034acfa2eeb77cb85a66b9', 'request_id': 'req-d016ac18-5a16-4836-882f-b2d3bce4e48a'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:54.794623 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-d016ac18-5a16-4836-882f-b2d3bce4e48a admin admin] The token cache is being invalidate because project 72d44c3f93034acfa2eeb77cb85a66b9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81508) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 00:55:54.795004 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-d016ac18-5a16-4836-882f-b2d3bce4e48a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:54.795858 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-d016ac18-5a16-4836-882f-b2d3bce4e48a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:54.801330 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 978/1934] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 00:55:54 2022] DELETE /identity/v3/projects/72d44c3f93034acfa2eeb77cb85a66b9 => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:55:54.802449 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98919e23-8c35-417f-9eac-9c131772d96a service neutron] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:54.813585 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98919e23-8c35-417f-9eac-9c131772d96a service neutron] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:54.825959 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d016ac18-5a16-4836-882f-b2d3bce4e48a admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:54.828062 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-d016ac18-5a16-4836-882f-b2d3bce4e48a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:54.852777 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:55:54.893724 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57878ff0-29ac-4c1b-917c-01c9014b7601 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:54.894560 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57878ff0-29ac-4c1b-917c-01c9014b7601 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:54.894930 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57878ff0-29ac-4c1b-917c-01c9014b7601 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:54.895296 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57878ff0-29ac-4c1b-917c-01c9014b7601 admin admin] PATH_INFO: `/v3/users/c0d3efa702504e9da6a14065c0619ab3` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:54.896600 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57878ff0-29ac-4c1b-917c-01c9014b7601 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c0d3efa702504e9da6a14065c0619ab3)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:54.898704 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57878ff0-29ac-4c1b-917c-01c9014b7601 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:54.924196 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d016ac18-5a16-4836-882f-b2d3bce4e48a admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:54.930175 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a07193de-b08e-4f50-92a7-e65491237742 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:54.930839 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a07193de-b08e-4f50-92a7-e65491237742 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:54.931168 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a07193de-b08e-4f50-92a7-e65491237742 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:54.931445 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a07193de-b08e-4f50-92a7-e65491237742 admin admin] PATH_INFO: `/v3/projects/25f494d3d7de482cb6131101a0cee139` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:54.932752 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a07193de-b08e-4f50-92a7-e65491237742 admin admin] RBAC: Authorizing `identity:delete_project(project_id=25f494d3d7de482cb6131101a0cee139)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:54.934767 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a07193de-b08e-4f50-92a7-e65491237742 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:54.963761 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-57878ff0-29ac-4c1b-917c-01c9014b7601 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c0d3efa702504e9da6a14065c0619ab3'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:54.975729 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=92,keystone:DELETE=8 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:55:54.981789 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-57878ff0-29ac-4c1b-917c-01c9014b7601 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c0d3efa702504e9da6a14065c0619ab3'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:54.990853 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-57878ff0-29ac-4c1b-917c-01c9014b7601 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c0d3efa702504e9da6a14065c0619ab3'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:54.995575 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-a07193de-b08e-4f50-92a7-e65491237742 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '25f494d3d7de482cb6131101a0cee139', 'request_id': 'req-a07193de-b08e-4f50-92a7-e65491237742'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:55.000570 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-57878ff0-29ac-4c1b-917c-01c9014b7601 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c0d3efa702504e9da6a14065c0619ab3'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:55.001346 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-57878ff0-29ac-4c1b-917c-01c9014b7601 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c0d3efa702504e9da6a14065c0619ab3'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:55.004233 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-a07193de-b08e-4f50-92a7-e65491237742 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '25f494d3d7de482cb6131101a0cee139', 'request_id': 'req-a07193de-b08e-4f50-92a7-e65491237742'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:55.007762 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 957/1935] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 00:55:54 2022] DELETE /identity/v3/users/c0d3efa702504e9da6a14065c0619ab3 => generated 0 bytes in 206 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:55:55.012991 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-a07193de-b08e-4f50-92a7-e65491237742 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '25f494d3d7de482cb6131101a0cee139', 'request_id': 'req-a07193de-b08e-4f50-92a7-e65491237742'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:55.028038 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-a07193de-b08e-4f50-92a7-e65491237742 admin admin] The token cache is being invalidate because project 25f494d3d7de482cb6131101a0cee139 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81508) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 00:55:55.028383 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-a07193de-b08e-4f50-92a7-e65491237742 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:55.029173 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-a07193de-b08e-4f50-92a7-e65491237742 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:55.031455 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 979/1936] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 00:55:54 2022] DELETE /identity/v3/projects/25f494d3d7de482cb6131101a0cee139 => generated 0 bytes in 206 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:55:55.035404 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57878ff0-29ac-4c1b-917c-01c9014b7601 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:55.036940 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-57878ff0-29ac-4c1b-917c-01c9014b7601 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:55.119674 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57878ff0-29ac-4c1b-917c-01c9014b7601 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:55.124949 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56b8c3fc-107b-4c7f-8908-852ace86841b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:55.125876 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56b8c3fc-107b-4c7f-8908-852ace86841b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:55.127225 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56b8c3fc-107b-4c7f-8908-852ace86841b admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:55.127474 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56b8c3fc-107b-4c7f-8908-852ace86841b admin admin] PATH_INFO: `/v3/users/e47fae5af1e94d9fb37a4bedbf7c0910` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:55.129457 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56b8c3fc-107b-4c7f-8908-852ace86841b admin admin] RBAC: Authorizing `identity:delete_user(user_id=e47fae5af1e94d9fb37a4bedbf7c0910)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:55.131578 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56b8c3fc-107b-4c7f-8908-852ace86841b admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:55.191133 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-56b8c3fc-107b-4c7f-8908-852ace86841b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e47fae5af1e94d9fb37a4bedbf7c0910'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:55.204286 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-56b8c3fc-107b-4c7f-8908-852ace86841b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e47fae5af1e94d9fb37a4bedbf7c0910'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:55.213442 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-56b8c3fc-107b-4c7f-8908-852ace86841b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e47fae5af1e94d9fb37a4bedbf7c0910'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:55.222040 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-56b8c3fc-107b-4c7f-8908-852ace86841b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e47fae5af1e94d9fb37a4bedbf7c0910'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:55.222815 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-56b8c3fc-107b-4c7f-8908-852ace86841b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e47fae5af1e94d9fb37a4bedbf7c0910'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:55.235283 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 958/1937] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 00:55:55 2022] DELETE /identity/v3/users/e47fae5af1e94d9fb37a4bedbf7c0910 => generated 0 bytes in 200 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:55:55.307409 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a07193de-b08e-4f50-92a7-e65491237742 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:55.309279 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-a07193de-b08e-4f50-92a7-e65491237742 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:55.380005 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a07193de-b08e-4f50-92a7-e65491237742 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:55.424530 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:55:55.469533 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46efa7ea-6f7b-47f1-b331-4118684f6d59 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:55.470347 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46efa7ea-6f7b-47f1-b331-4118684f6d59 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:55.470718 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46efa7ea-6f7b-47f1-b331-4118684f6d59 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:55.471084 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46efa7ea-6f7b-47f1-b331-4118684f6d59 admin admin] PATH_INFO: `/v3/projects/c062f3365c2f496290dd2f383776476c` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:55.472324 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46efa7ea-6f7b-47f1-b331-4118684f6d59 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c062f3365c2f496290dd2f383776476c)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:55.474357 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46efa7ea-6f7b-47f1-b331-4118684f6d59 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:55.534867 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-46efa7ea-6f7b-47f1-b331-4118684f6d59 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c062f3365c2f496290dd2f383776476c', 'request_id': 'req-46efa7ea-6f7b-47f1-b331-4118684f6d59'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:55.541531 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-46efa7ea-6f7b-47f1-b331-4118684f6d59 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c062f3365c2f496290dd2f383776476c', 'request_id': 'req-46efa7ea-6f7b-47f1-b331-4118684f6d59'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:55.553858 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-46efa7ea-6f7b-47f1-b331-4118684f6d59 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c062f3365c2f496290dd2f383776476c', 'request_id': 'req-46efa7ea-6f7b-47f1-b331-4118684f6d59'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:55.554239 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56b8c3fc-107b-4c7f-8908-852ace86841b admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:55.559407 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-56b8c3fc-107b-4c7f-8908-852ace86841b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:55.569158 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-46efa7ea-6f7b-47f1-b331-4118684f6d59 admin admin] The token cache is being invalidate because project c062f3365c2f496290dd2f383776476c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81508) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 00:55:55.569716 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-46efa7ea-6f7b-47f1-b331-4118684f6d59 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:55.570488 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-46efa7ea-6f7b-47f1-b331-4118684f6d59 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:55.572559 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 980/1938] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 00:55:55 2022] DELETE /identity/v3/projects/c062f3365c2f496290dd2f383776476c => generated 0 bytes in 266 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:55:55.577524 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46efa7ea-6f7b-47f1-b331-4118684f6d59 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:55.579556 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-46efa7ea-6f7b-47f1-b331-4118684f6d59 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:55.580225 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56b8c3fc-107b-4c7f-8908-852ace86841b admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:55.651585 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:55:55.660109 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a5f847d-3e65-4d4f-add1-525605c12209 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:55.661009 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a5f847d-3e65-4d4f-add1-525605c12209 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:55.661917 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a5f847d-3e65-4d4f-add1-525605c12209 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:55.662206 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a5f847d-3e65-4d4f-add1-525605c12209 admin admin] PATH_INFO: `/v3/projects/a26f0057529a455c9c5b3bfb363c4b4e` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:55.663904 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a5f847d-3e65-4d4f-add1-525605c12209 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a26f0057529a455c9c5b3bfb363c4b4e)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:55.667053 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a5f847d-3e65-4d4f-add1-525605c12209 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:55.676026 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46efa7ea-6f7b-47f1-b331-4118684f6d59 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:55.683336 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71ec86f7-c40e-44dc-8316-22056b0f1717 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:55.684271 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ec86f7-c40e-44dc-8316-22056b0f1717 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:55.684750 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ec86f7-c40e-44dc-8316-22056b0f1717 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:55.685075 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ec86f7-c40e-44dc-8316-22056b0f1717 admin admin] PATH_INFO: `/v3/users/dd25ccae1bcc4099bddee8a1259317a5` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:55.686281 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71ec86f7-c40e-44dc-8316-22056b0f1717 admin admin] RBAC: Authorizing `identity:delete_user(user_id=dd25ccae1bcc4099bddee8a1259317a5)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:55.688617 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71ec86f7-c40e-44dc-8316-22056b0f1717 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:55.727738 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-0a5f847d-3e65-4d4f-add1-525605c12209 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a26f0057529a455c9c5b3bfb363c4b4e', 'request_id': 'req-0a5f847d-3e65-4d4f-add1-525605c12209'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:55.736384 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-0a5f847d-3e65-4d4f-add1-525605c12209 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a26f0057529a455c9c5b3bfb363c4b4e', 'request_id': 'req-0a5f847d-3e65-4d4f-add1-525605c12209'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:55.742877 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-0a5f847d-3e65-4d4f-add1-525605c12209 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a26f0057529a455c9c5b3bfb363c4b4e', 'request_id': 'req-0a5f847d-3e65-4d4f-add1-525605c12209'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:55.753316 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-71ec86f7-c40e-44dc-8316-22056b0f1717 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'dd25ccae1bcc4099bddee8a1259317a5'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:55.754333 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-71ec86f7-c40e-44dc-8316-22056b0f1717 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'dd25ccae1bcc4099bddee8a1259317a5'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:55.757152 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-0a5f847d-3e65-4d4f-add1-525605c12209 admin admin] The token cache is being invalidate because project a26f0057529a455c9c5b3bfb363c4b4e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81507) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 00:55:55.757447 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-0a5f847d-3e65-4d4f-add1-525605c12209 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:55.758074 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-0a5f847d-3e65-4d4f-add1-525605c12209 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:55.759991 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 959/1939] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 00:55:55 2022] DELETE /identity/v3/projects/a26f0057529a455c9c5b3bfb363c4b4e => generated 0 bytes in 209 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:55:55.763573 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-71ec86f7-c40e-44dc-8316-22056b0f1717 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'dd25ccae1bcc4099bddee8a1259317a5'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:55.763994 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a5f847d-3e65-4d4f-add1-525605c12209 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:55.765606 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-0a5f847d-3e65-4d4f-add1-525605c12209 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:55.773787 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-71ec86f7-c40e-44dc-8316-22056b0f1717 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'dd25ccae1bcc4099bddee8a1259317a5'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:55.774532 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-71ec86f7-c40e-44dc-8316-22056b0f1717 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'dd25ccae1bcc4099bddee8a1259317a5'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:55.799096 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:55:55.818971 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 981/1940] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 00:55:55 2022] DELETE /identity/v3/users/dd25ccae1bcc4099bddee8a1259317a5 => generated 0 bytes in 242 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:55:55.883645 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a5f847d-3e65-4d4f-add1-525605c12209 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:55.964041 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fffa018-f4c7-4422-916a-021235b65e1e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:55.964744 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fffa018-f4c7-4422-916a-021235b65e1e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:55.965109 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fffa018-f4c7-4422-916a-021235b65e1e admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:55.965413 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fffa018-f4c7-4422-916a-021235b65e1e admin admin] PATH_INFO: `/v3/users/31c6473ee616414e9aef96b474e13e45` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:55.966642 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fffa018-f4c7-4422-916a-021235b65e1e admin admin] RBAC: Authorizing `identity:delete_user(user_id=31c6473ee616414e9aef96b474e13e45)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:55.968725 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fffa018-f4c7-4422-916a-021235b65e1e admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:56.021642 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:55:56.042180 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-5fffa018-f4c7-4422-916a-021235b65e1e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '31c6473ee616414e9aef96b474e13e45'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:56.049295 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71ec86f7-c40e-44dc-8316-22056b0f1717 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:56.051775 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-71ec86f7-c40e-44dc-8316-22056b0f1717 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:56.065196 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-5fffa018-f4c7-4422-916a-021235b65e1e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '31c6473ee616414e9aef96b474e13e45'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:56.078816 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-5fffa018-f4c7-4422-916a-021235b65e1e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '31c6473ee616414e9aef96b474e13e45'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:56.083887 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71ec86f7-c40e-44dc-8316-22056b0f1717 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:56.091936 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e85d355-4a02-4756-9ff3-eb4c8fa4b2d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:56.092555 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-5fffa018-f4c7-4422-916a-021235b65e1e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '31c6473ee616414e9aef96b474e13e45'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:56.093162 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e85d355-4a02-4756-9ff3-eb4c8fa4b2d4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:56.093661 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e85d355-4a02-4756-9ff3-eb4c8fa4b2d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:56.093787 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-5fffa018-f4c7-4422-916a-021235b65e1e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '31c6473ee616414e9aef96b474e13e45'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:56.094227 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e85d355-4a02-4756-9ff3-eb4c8fa4b2d4 admin admin] PATH_INFO: `/v3/projects/9d823c42c76544638a09799689a218eb` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:56.095671 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e85d355-4a02-4756-9ff3-eb4c8fa4b2d4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9d823c42c76544638a09799689a218eb)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:56.099051 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e85d355-4a02-4756-9ff3-eb4c8fa4b2d4 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:56.103861 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 960/1941] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 00:55:55 2022] DELETE /identity/v3/users/31c6473ee616414e9aef96b474e13e45 => generated 0 bytes in 337 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 28 00:55:56.176347 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-6e85d355-4a02-4756-9ff3-eb4c8fa4b2d4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9d823c42c76544638a09799689a218eb', 'request_id': 'req-6e85d355-4a02-4756-9ff3-eb4c8fa4b2d4'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:56.183200 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-6e85d355-4a02-4756-9ff3-eb4c8fa4b2d4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9d823c42c76544638a09799689a218eb', 'request_id': 'req-6e85d355-4a02-4756-9ff3-eb4c8fa4b2d4'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:56.189931 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-6e85d355-4a02-4756-9ff3-eb4c8fa4b2d4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9d823c42c76544638a09799689a218eb', 'request_id': 'req-6e85d355-4a02-4756-9ff3-eb4c8fa4b2d4'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:56.209315 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-6e85d355-4a02-4756-9ff3-eb4c8fa4b2d4 admin admin] The token cache is being invalidate because project 9d823c42c76544638a09799689a218eb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81508) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 00:55:56.209889 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-6e85d355-4a02-4756-9ff3-eb4c8fa4b2d4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:56.210912 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-6e85d355-4a02-4756-9ff3-eb4c8fa4b2d4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:56.213232 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 982/1942] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 00:55:56 2022] DELETE /identity/v3/projects/9d823c42c76544638a09799689a218eb => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 28 00:55:56.219879 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d50d551-752a-46e3-9fac-6f6f1a33c44e None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:56.220169 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d50d551-752a-46e3-9fac-6f6f1a33c44e None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:56.220445 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d50d551-752a-46e3-9fac-6f6f1a33c44e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:56.260185 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-2d50d551-752a-46e3-9fac-6f6f1a33c44e None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:55:56.262390 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-2d50d551-752a-46e3-9fac-6f6f1a33c44e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:56.336114 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e85d355-4a02-4756-9ff3-eb4c8fa4b2d4 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:56.338448 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-6e85d355-4a02-4756-9ff3-eb4c8fa4b2d4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:56.371045 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 961/1943] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:55:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:55:56.379930 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3acef590-2df8-44a3-abe8-447d3b56cc42 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:56.380220 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3acef590-2df8-44a3-abe8-447d3b56cc42 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:56.380496 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3acef590-2df8-44a3-abe8-447d3b56cc42 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:56.401929 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-3acef590-2df8-44a3-abe8-447d3b56cc42 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:55:56.403168 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-3acef590-2df8-44a3-abe8-447d3b56cc42 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:56.417328 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 962/1944] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:55:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 28 00:55:56.418609 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3acef590-2df8-44a3-abe8-447d3b56cc42 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:56.430731 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3acef590-2df8-44a3-abe8-447d3b56cc42 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:56.436030 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd336678-d4cc-4358-99fc-63921ac97304 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:56.436678 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd336678-d4cc-4358-99fc-63921ac97304 admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:56.437050 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd336678-d4cc-4358-99fc-63921ac97304 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:56.437328 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd336678-d4cc-4358-99fc-63921ac97304 admin admin] PATH_INFO: `/v3/domains` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:56.438203 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd336678-d4cc-4358-99fc-63921ac97304 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:55:56.438516 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd336678-d4cc-4358-99fc-63921ac97304 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:56.440230 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd336678-d4cc-4358-99fc-63921ac97304 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:56.460199 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 963/1945] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Aug 28 00:55:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:55:56.461241 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd336678-d4cc-4358-99fc-63921ac97304 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:56.488128 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd336678-d4cc-4358-99fc-63921ac97304 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:56.495007 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b692fbfc-e3cc-45e9-90f9-6b542a275e7e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:56.501383 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b692fbfc-e3cc-45e9-90f9-6b542a275e7e admin admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:56.501714 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b692fbfc-e3cc-45e9-90f9-6b542a275e7e admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:56.501992 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b692fbfc-e3cc-45e9-90f9-6b542a275e7e admin admin] PATH_INFO: `/v3/projects` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:56.503280 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b692fbfc-e3cc-45e9-90f9-6b542a275e7e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:56.505393 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b692fbfc-e3cc-45e9-90f9-6b542a275e7e admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:56.509522 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e85d355-4a02-4756-9ff3-eb4c8fa4b2d4 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:56.515548 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e84b3e9-490b-407b-9f17-7707ee1d87a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:56.516367 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e84b3e9-490b-407b-9f17-7707ee1d87a3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:56.516790 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e84b3e9-490b-407b-9f17-7707ee1d87a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:56.517188 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e84b3e9-490b-407b-9f17-7707ee1d87a3 admin admin] PATH_INFO: `/v3/projects/dc992a50f7e74e77af91babe2d17f5ad` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:56.518637 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e84b3e9-490b-407b-9f17-7707ee1d87a3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=dc992a50f7e74e77af91babe2d17f5ad)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:56.521539 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e84b3e9-490b-407b-9f17-7707ee1d87a3 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:56.558715 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 964/1946] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 00:55:56 2022] POST /identity/v3/projects => generated 361 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:55:56.562638 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b692fbfc-e3cc-45e9-90f9-6b542a275e7e admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:56.599821 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-5e84b3e9-490b-407b-9f17-7707ee1d87a3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'dc992a50f7e74e77af91babe2d17f5ad', 'request_id': 'req-5e84b3e9-490b-407b-9f17-7707ee1d87a3'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:56.608059 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-5e84b3e9-490b-407b-9f17-7707ee1d87a3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'dc992a50f7e74e77af91babe2d17f5ad', 'request_id': 'req-5e84b3e9-490b-407b-9f17-7707ee1d87a3'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:56.616485 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-5e84b3e9-490b-407b-9f17-7707ee1d87a3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'dc992a50f7e74e77af91babe2d17f5ad', 'request_id': 'req-5e84b3e9-490b-407b-9f17-7707ee1d87a3'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:56.636129 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-5e84b3e9-490b-407b-9f17-7707ee1d87a3 admin admin] The token cache is being invalidate because project dc992a50f7e74e77af91babe2d17f5ad was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81508) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 00:55:56.636734 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-5e84b3e9-490b-407b-9f17-7707ee1d87a3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:56.637966 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-5e84b3e9-490b-407b-9f17-7707ee1d87a3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:56.641121 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 983/1947] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 00:55:56 2022] DELETE /identity/v3/projects/dc992a50f7e74e77af91babe2d17f5ad => generated 0 bytes in 305 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 28 00:55:56.646543 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ac48758-e0ed-4164-a5f2-52d693adca69 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:56.647466 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ac48758-e0ed-4164-a5f2-52d693adca69 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:56.648311 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ac48758-e0ed-4164-a5f2-52d693adca69 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:56.662451 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b692fbfc-e3cc-45e9-90f9-6b542a275e7e admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:56.679539 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=93,keystone:DELETE=5,keystone:INSERT=2 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:55:56.680876 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-7ac48758-e0ed-4164-a5f2-52d693adca69 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:55:56.682901 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-7ac48758-e0ed-4164-a5f2-52d693adca69 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:56.691303 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:55:56.761567 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61260725-0b4c-498b-958e-ff6561cd36be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:56.762205 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61260725-0b4c-498b-958e-ff6561cd36be admin admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:56.762483 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61260725-0b4c-498b-958e-ff6561cd36be admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:56.762753 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61260725-0b4c-498b-958e-ff6561cd36be admin admin] PATH_INFO: `/v3/users` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:56.763484 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61260725-0b4c-498b-958e-ff6561cd36be admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:56.765584 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61260725-0b4c-498b-958e-ff6561cd36be admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:56.785206 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 984/1948] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:55:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:55:56.785206 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ac48758-e0ed-4164-a5f2-52d693adca69 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:56.786473 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-7ac48758-e0ed-4164-a5f2-52d693adca69 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:56.793299 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 965/1949] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 00:55:56 2022] POST /identity/v3/users => generated 281 bytes in 231 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:55:56.794890 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f368dd2c-2a3b-4e52-ba05-81e8d4ac51ed None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:56.795412 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f368dd2c-2a3b-4e52-ba05-81e8d4ac51ed None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:56.795889 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f368dd2c-2a3b-4e52-ba05-81e8d4ac51ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:56.814573 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ac48758-e0ed-4164-a5f2-52d693adca69 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:56.818057 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-f368dd2c-2a3b-4e52-ba05-81e8d4ac51ed None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:55:56.819384 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-f368dd2c-2a3b-4e52-ba05-81e8d4ac51ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:56.821999 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42993560-7dca-411c-8901-fdc779748e05 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:56.823011 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42993560-7dca-411c-8901-fdc779748e05 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:56.823433 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42993560-7dca-411c-8901-fdc779748e05 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:56.824200 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42993560-7dca-411c-8901-fdc779748e05 admin admin] PATH_INFO: `/v3/projects/ec15f147ec06441d99263f79a527958e` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:56.825590 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42993560-7dca-411c-8901-fdc779748e05 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ec15f147ec06441d99263f79a527958e)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:56.828027 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42993560-7dca-411c-8901-fdc779748e05 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:56.831807 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 966/1950] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:55:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:55:56.832469 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f368dd2c-2a3b-4e52-ba05-81e8d4ac51ed None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:56.833787 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-f368dd2c-2a3b-4e52-ba05-81e8d4ac51ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:56.848938 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f368dd2c-2a3b-4e52-ba05-81e8d4ac51ed None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:56.854161 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-593aec69-ddf6-43d8-97eb-39aa96ab1578 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:56.854780 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-593aec69-ddf6-43d8-97eb-39aa96ab1578 admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:56.855057 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-593aec69-ddf6-43d8-97eb-39aa96ab1578 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:56.855323 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-593aec69-ddf6-43d8-97eb-39aa96ab1578 admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:56.856031 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-593aec69-ddf6-43d8-97eb-39aa96ab1578 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:56.857687 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-593aec69-ddf6-43d8-97eb-39aa96ab1578 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:56.873198 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 967/1951] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:55:56 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 00:55:56.873198 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-593aec69-ddf6-43d8-97eb-39aa96ab1578 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:56.885591 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-593aec69-ddf6-43d8-97eb-39aa96ab1578 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:56.891881 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-42993560-7dca-411c-8901-fdc779748e05 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ec15f147ec06441d99263f79a527958e', 'request_id': 'req-42993560-7dca-411c-8901-fdc779748e05'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:56.898664 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-42993560-7dca-411c-8901-fdc779748e05 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ec15f147ec06441d99263f79a527958e', 'request_id': 'req-42993560-7dca-411c-8901-fdc779748e05'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:56.911524 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-42993560-7dca-411c-8901-fdc779748e05 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ec15f147ec06441d99263f79a527958e', 'request_id': 'req-42993560-7dca-411c-8901-fdc779748e05'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:56.930358 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-42993560-7dca-411c-8901-fdc779748e05 admin admin] The token cache is being invalidate because project ec15f147ec06441d99263f79a527958e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81508) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 00:55:56.931264 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-42993560-7dca-411c-8901-fdc779748e05 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:56.932129 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-42993560-7dca-411c-8901-fdc779748e05 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:56.934085 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 985/1952] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 00:55:56 2022] DELETE /identity/v3/projects/ec15f147ec06441d99263f79a527958e => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:55:56.935089 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42993560-7dca-411c-8901-fdc779748e05 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:56.937365 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-42993560-7dca-411c-8901-fdc779748e05 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:56.972137 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-077d4b55-fa3c-44d4-bd0b-a8b493011a85 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:56.972829 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-077d4b55-fa3c-44d4-bd0b-a8b493011a85 admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:56.973175 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-077d4b55-fa3c-44d4-bd0b-a8b493011a85 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:56.973446 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-077d4b55-fa3c-44d4-bd0b-a8b493011a85 admin admin] PATH_INFO: `/v3/domains` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:56.974297 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-077d4b55-fa3c-44d4-bd0b-a8b493011a85 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:55:56.974991 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-077d4b55-fa3c-44d4-bd0b-a8b493011a85 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:56.976748 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-077d4b55-fa3c-44d4-bd0b-a8b493011a85 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:56.991558 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 968/1953] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Aug 28 00:55:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 119 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:55:56.993092 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77745f96-a6ab-4597-85ac-f4e4711acd1c None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:56.993378 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77745f96-a6ab-4597-85ac-f4e4711acd1c None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:56.993649 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77745f96-a6ab-4597-85ac-f4e4711acd1c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:57.019543 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-77745f96-a6ab-4597-85ac-f4e4711acd1c None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:55:57.021711 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-77745f96-a6ab-4597-85ac-f4e4711acd1c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:57.024774 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42993560-7dca-411c-8901-fdc779748e05 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:57.029763 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ddaaacf-d6f7-4a15-a884-53552f52c011 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:57.030779 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 969/1954] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:55:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:55:57.031807 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77745f96-a6ab-4597-85ac-f4e4711acd1c None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:57.033238 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-77745f96-a6ab-4597-85ac-f4e4711acd1c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:57.035053 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ddaaacf-d6f7-4a15-a884-53552f52c011 admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:57.035439 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ddaaacf-d6f7-4a15-a884-53552f52c011 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:57.035730 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ddaaacf-d6f7-4a15-a884-53552f52c011 admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:57.036417 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ddaaacf-d6f7-4a15-a884-53552f52c011 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:57.038402 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ddaaacf-d6f7-4a15-a884-53552f52c011 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:57.047306 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77745f96-a6ab-4597-85ac-f4e4711acd1c None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:57.049870 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 986/1955] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:55:56 2022] GET /identity/v3/roles => generated 1420 bytes in 115 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 00:55:57.051361 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30c13faf-18d9-4440-9e55-fdb64e466884 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:57.051361 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30c13faf-18d9-4440-9e55-fdb64e466884 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:57.051361 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30c13faf-18d9-4440-9e55-fdb64e466884 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:57.055793 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59a5ba33-4403-4b4b-9043-c314b07d6178 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:57.056534 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a5ba33-4403-4b4b-9043-c314b07d6178 admin admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:57.056876 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a5ba33-4403-4b4b-9043-c314b07d6178 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:57.057236 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a5ba33-4403-4b4b-9043-c314b07d6178 admin admin] PATH_INFO: `/v3/projects` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:57.057950 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59a5ba33-4403-4b4b-9043-c314b07d6178 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:57.060134 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59a5ba33-4403-4b4b-9043-c314b07d6178 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:57.083161 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-30c13faf-18d9-4440-9e55-fdb64e466884 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:55:57.083798 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-30c13faf-18d9-4440-9e55-fdb64e466884 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:57.094821 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 987/1956] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:55:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 45 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:55:57.098989 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30c13faf-18d9-4440-9e55-fdb64e466884 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:57.100853 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 970/1957] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 00:55:57 2022] POST /identity/v3/projects => generated 383 bytes in 69 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:55:57.101757 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59a5ba33-4403-4b4b-9043-c314b07d6178 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:57.135853 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30c13faf-18d9-4440-9e55-fdb64e466884 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:57.168653 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:55:57.173521 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:55:57.182132 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59a5ba33-4403-4b4b-9043-c314b07d6178 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:57.188557 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6400c27e-b8f5-423d-a154-6f132bbe6ba5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:57.189350 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6400c27e-b8f5-423d-a154-6f132bbe6ba5 admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:57.189686 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6400c27e-b8f5-423d-a154-6f132bbe6ba5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:57.190007 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6400c27e-b8f5-423d-a154-6f132bbe6ba5 admin admin] PATH_INFO: `/v3/domains` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:57.190737 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6400c27e-b8f5-423d-a154-6f132bbe6ba5 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:55:57.191097 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6400c27e-b8f5-423d-a154-6f132bbe6ba5 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:57.193605 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6400c27e-b8f5-423d-a154-6f132bbe6ba5 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:57.210677 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 971/1958] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Aug 28 00:55:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 109 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:55:57.211478 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6400c27e-b8f5-423d-a154-6f132bbe6ba5 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:57.226664 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6400c27e-b8f5-423d-a154-6f132bbe6ba5 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:57.234204 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cc95857-f17b-46f9-9324-e37bb5844dc1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:57.234917 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc95857-f17b-46f9-9324-e37bb5844dc1 admin admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:57.235254 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc95857-f17b-46f9-9324-e37bb5844dc1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:57.235590 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc95857-f17b-46f9-9324-e37bb5844dc1 admin admin] PATH_INFO: `/v3/users` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:57.236304 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cc95857-f17b-46f9-9324-e37bb5844dc1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:57.238416 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cc95857-f17b-46f9-9324-e37bb5844dc1 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:57.248869 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6469899c-479d-4304-8b76-a8023b557fc3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:57.249931 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6469899c-479d-4304-8b76-a8023b557fc3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:57.250053 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6469899c-479d-4304-8b76-a8023b557fc3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:57.250351 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6469899c-479d-4304-8b76-a8023b557fc3 admin admin] PATH_INFO: `/v3/projects/38f4fef6151241d494acc66a152703f3/users/c10e2ad4f9e0462ebad5d7e0c3222b68/roles/0487ef7e943e4701915cb8ce0f2899d7` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:57.267371 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 972/1959] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 00:55:57 2022] POST /identity/v3/users => generated 292 bytes in 56 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:55:57.268441 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cc95857-f17b-46f9-9324-e37bb5844dc1 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:57.274269 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6469899c-479d-4304-8b76-a8023b557fc3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=38f4fef6151241d494acc66a152703f3, user_id=c10e2ad4f9e0462ebad5d7e0c3222b68, role_id=0487ef7e943e4701915cb8ce0f2899d7)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:57.277111 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6469899c-479d-4304-8b76-a8023b557fc3 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:57.285500 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cc95857-f17b-46f9-9324-e37bb5844dc1 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:57.294452 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d8c33e6-f5d0-4eb0-8c5e-a38fb278edce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:57.295239 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d8c33e6-f5d0-4eb0-8c5e-a38fb278edce admin admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:57.295525 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d8c33e6-f5d0-4eb0-8c5e-a38fb278edce admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:57.295796 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d8c33e6-f5d0-4eb0-8c5e-a38fb278edce admin admin] PATH_INFO: `/v3/projects` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:57.296506 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d8c33e6-f5d0-4eb0-8c5e-a38fb278edce admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:57.298024 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 988/1960] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 00:55:57 2022] PUT /identity/v3/projects/38f4fef6151241d494acc66a152703f3/users/c10e2ad4f9e0462ebad5d7e0c3222b68/roles/0487ef7e943e4701915cb8ce0f2899d7 => generated 0 bytes in 202 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:55:57.298475 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d8c33e6-f5d0-4eb0-8c5e-a38fb278edce admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:57.309476 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6469899c-479d-4304-8b76-a8023b557fc3 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:57.353095 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 973/1961] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 00:55:57 2022] POST /identity/v3/projects => generated 375 bytes in 82 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:55:57.353095 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62477024-34bb-4b07-85af-d241a83ff0c9 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:57.353095 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62477024-34bb-4b07-85af-d241a83ff0c9 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:57.353095 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62477024-34bb-4b07-85af-d241a83ff0c9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:57.396536 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-62477024-34bb-4b07-85af-d241a83ff0c9 None None] MFA Rules not processed for user `c10e2ad4f9e0462ebad5d7e0c3222b68`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:55:57.398759 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-62477024-34bb-4b07-85af-d241a83ff0c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:57.441303 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6469899c-479d-4304-8b76-a8023b557fc3 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:57.503096 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 974/1962] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:55:57 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 152 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:55:57.504202 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62477024-34bb-4b07-85af-d241a83ff0c9 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:57.635408 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e184e0d1-f709-4cc8-ba42-798c3afb7aae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:57.637831 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e184e0d1-f709-4cc8-ba42-798c3afb7aae admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:57.638125 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e184e0d1-f709-4cc8-ba42-798c3afb7aae admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:57.640610 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e184e0d1-f709-4cc8-ba42-798c3afb7aae admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:57.642829 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e184e0d1-f709-4cc8-ba42-798c3afb7aae admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:57.644868 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e184e0d1-f709-4cc8-ba42-798c3afb7aae admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:57.661959 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 989/1963] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:55:57 2022] GET /identity/v3/roles => generated 1420 bytes in 363 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 00:55:57.662306 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e184e0d1-f709-4cc8-ba42-798c3afb7aae admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:57.664036 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-e184e0d1-f709-4cc8-ba42-798c3afb7aae admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:57.666634 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62477024-34bb-4b07-85af-d241a83ff0c9 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:57.672471 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9aac400-bd7d-4acf-95ba-82267349876b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:57.673322 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9aac400-bd7d-4acf-95ba-82267349876b admin admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:57.673868 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9aac400-bd7d-4acf-95ba-82267349876b admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:57.674292 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9aac400-bd7d-4acf-95ba-82267349876b admin admin] PATH_INFO: `/v3/users` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:57.675114 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9aac400-bd7d-4acf-95ba-82267349876b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:57.677858 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9aac400-bd7d-4acf-95ba-82267349876b admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:57.706461 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 975/1964] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 00:55:57 2022] POST /identity/v3/users => generated 288 bytes in 201 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:55:57.711338 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9aac400-bd7d-4acf-95ba-82267349876b admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:57.732515 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9aac400-bd7d-4acf-95ba-82267349876b admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:57.738408 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f00caeb-6456-4cb6-9b7d-979e3ca3e43f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:57.739177 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f00caeb-6456-4cb6-9b7d-979e3ca3e43f admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:57.739458 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f00caeb-6456-4cb6-9b7d-979e3ca3e43f admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:57.739903 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f00caeb-6456-4cb6-9b7d-979e3ca3e43f admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:57.740729 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f00caeb-6456-4cb6-9b7d-979e3ca3e43f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:57.742567 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f00caeb-6456-4cb6-9b7d-979e3ca3e43f admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:57.749614 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:55:57.755166 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e184e0d1-f709-4cc8-ba42-798c3afb7aae admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:57.758778 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 976/1965] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:55:57 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 00:55:57.758778 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f00caeb-6456-4cb6-9b7d-979e3ca3e43f admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:57.765966 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8961daab-53be-4028-9705-92fc85d57a8a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:57.766693 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8961daab-53be-4028-9705-92fc85d57a8a service cinder] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:57.767061 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8961daab-53be-4028-9705-92fc85d57a8a service cinder] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:57.767324 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8961daab-53be-4028-9705-92fc85d57a8a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:57.778335 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8961daab-53be-4028-9705-92fc85d57a8a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:57.779383 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f00caeb-6456-4cb6-9b7d-979e3ca3e43f admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:57.779832 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8961daab-53be-4028-9705-92fc85d57a8a service cinder] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:57.784611 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5655ee6a-c271-4a01-80be-4b413c978dd8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:57.785430 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5655ee6a-c271-4a01-80be-4b413c978dd8 admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:57.785808 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5655ee6a-c271-4a01-80be-4b413c978dd8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:57.786181 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5655ee6a-c271-4a01-80be-4b413c978dd8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:57.786929 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5655ee6a-c271-4a01-80be-4b413c978dd8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:57.788637 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5655ee6a-c271-4a01-80be-4b413c978dd8 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:57.791358 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 990/1966] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:55:57 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 129 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:55:57.792392 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8961daab-53be-4028-9705-92fc85d57a8a service cinder] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:57.806522 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 977/1967] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:55:57 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 00:55:57.808363 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5655ee6a-c271-4a01-80be-4b413c978dd8 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:57.815086 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8961daab-53be-4028-9705-92fc85d57a8a service cinder] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:57.819947 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9a16490-61fa-4ee3-b88c-f11a4cd9382e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:57.820636 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a16490-61fa-4ee3-b88c-f11a4cd9382e admin admin] REQUEST_METHOD: `PUT` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:57.821184 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5655ee6a-c271-4a01-80be-4b413c978dd8 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:57.821301 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a16490-61fa-4ee3-b88c-f11a4cd9382e admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:57.821653 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a16490-61fa-4ee3-b88c-f11a4cd9382e admin admin] PATH_INFO: `/v3/projects/6e52a2d256554d84b23f97c15bcc06a4/users/f8f56a6153f841de8612b40f65d8db69/roles/0487ef7e943e4701915cb8ce0f2899d7` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:57.828397 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-944ecadd-92e3-4212-993f-fdcc0fa56565 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:57.829232 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-944ecadd-92e3-4212-993f-fdcc0fa56565 admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:57.829618 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-944ecadd-92e3-4212-993f-fdcc0fa56565 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:57.829991 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-944ecadd-92e3-4212-993f-fdcc0fa56565 admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:57.830838 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-944ecadd-92e3-4212-993f-fdcc0fa56565 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:57.832588 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-944ecadd-92e3-4212-993f-fdcc0fa56565 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:57.835271 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:55:57.843917 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9a16490-61fa-4ee3-b88c-f11a4cd9382e admin admin] RBAC: Authorizing `identity:create_grant(project_id=6e52a2d256554d84b23f97c15bcc06a4, user_id=f8f56a6153f841de8612b40f65d8db69, role_id=0487ef7e943e4701915cb8ce0f2899d7)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:57.847193 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9a16490-61fa-4ee3-b88c-f11a4cd9382e admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:57.852524 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 978/1968] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:55:57 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 00:55:57.853636 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-944ecadd-92e3-4212-993f-fdcc0fa56565 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:57.865520 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-944ecadd-92e3-4212-993f-fdcc0fa56565 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:57.880995 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 991/1969] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 00:55:57 2022] PUT /identity/v3/projects/6e52a2d256554d84b23f97c15bcc06a4/users/f8f56a6153f841de8612b40f65d8db69/roles/0487ef7e943e4701915cb8ce0f2899d7 => generated 0 bytes in 77 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:55:57.880995 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9a16490-61fa-4ee3-b88c-f11a4cd9382e admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:57.959008 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9a16490-61fa-4ee3-b88c-f11a4cd9382e admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:57.964559 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4d11c30-0169-4828-8a1b-c9eed25850db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:57.965308 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d11c30-0169-4828-8a1b-c9eed25850db admin admin] REQUEST_METHOD: `PUT` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:57.965586 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d11c30-0169-4828-8a1b-c9eed25850db admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:57.965857 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d11c30-0169-4828-8a1b-c9eed25850db admin admin] PATH_INFO: `/v3/projects/66dbaa2c60594d73ac6fb66012c6061b/users/89aa67ee7dff4ad5beb433c399dc1b7f/roles/0487ef7e943e4701915cb8ce0f2899d7` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:57.976593 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be27acab-c375-4ae6-839a-4f87ee27266f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:57.977860 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be27acab-c375-4ae6-839a-4f87ee27266f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:57.978357 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be27acab-c375-4ae6-839a-4f87ee27266f admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:57.979652 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be27acab-c375-4ae6-839a-4f87ee27266f admin admin] PATH_INFO: `/v3/users/c10e2ad4f9e0462ebad5d7e0c3222b68` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:57.981166 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be27acab-c375-4ae6-839a-4f87ee27266f admin admin] RBAC: Authorizing `identity:delete_user(user_id=c10e2ad4f9e0462ebad5d7e0c3222b68)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:57.983288 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be27acab-c375-4ae6-839a-4f87ee27266f admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:58.002677 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4d11c30-0169-4828-8a1b-c9eed25850db admin admin] RBAC: Authorizing `identity:create_grant(project_id=66dbaa2c60594d73ac6fb66012c6061b, user_id=89aa67ee7dff4ad5beb433c399dc1b7f, role_id=0487ef7e943e4701915cb8ce0f2899d7)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:58.005444 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4d11c30-0169-4828-8a1b-c9eed25850db admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:58.026086 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 992/1970] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 00:55:57 2022] PUT /identity/v3/projects/66dbaa2c60594d73ac6fb66012c6061b/users/89aa67ee7dff4ad5beb433c399dc1b7f/roles/0487ef7e943e4701915cb8ce0f2899d7 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:55:58.033235 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7003f17a-7e87-47a1-8d4e-7842936f7c1d None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:58.033235 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7003f17a-7e87-47a1-8d4e-7842936f7c1d None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:58.033235 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7003f17a-7e87-47a1-8d4e-7842936f7c1d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:58.078164 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-be27acab-c375-4ae6-839a-4f87ee27266f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c10e2ad4f9e0462ebad5d7e0c3222b68'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:58.082678 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-7003f17a-7e87-47a1-8d4e-7842936f7c1d None None] MFA Rules not processed for user `f8f56a6153f841de8612b40f65d8db69`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:55:58.083980 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-7003f17a-7e87-47a1-8d4e-7842936f7c1d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:58.103581 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-be27acab-c375-4ae6-839a-4f87ee27266f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c10e2ad4f9e0462ebad5d7e0c3222b68'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:58.137637 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-be27acab-c375-4ae6-839a-4f87ee27266f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c10e2ad4f9e0462ebad5d7e0c3222b68'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:58.164600 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-be27acab-c375-4ae6-839a-4f87ee27266f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c10e2ad4f9e0462ebad5d7e0c3222b68'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:58.165566 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-be27acab-c375-4ae6-839a-4f87ee27266f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c10e2ad4f9e0462ebad5d7e0c3222b68'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:58.177801 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 979/1971] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 00:55:57 2022] DELETE /identity/v3/users/c10e2ad4f9e0462ebad5d7e0c3222b68 => generated 0 bytes in 321 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:55:58.177801 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2589718-5d7f-4981-a975-1dab5fa233e3 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:58.177801 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2589718-5d7f-4981-a975-1dab5fa233e3 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:58.177801 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2589718-5d7f-4981-a975-1dab5fa233e3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:58.206912 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 993/1972] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:55:58 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:55:58.210114 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a3b851a-25bd-40e8-9b78-83188068fe94 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:58.211185 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a3b851a-25bd-40e8-9b78-83188068fe94 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:58.212335 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a3b851a-25bd-40e8-9b78-83188068fe94 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:58.212584 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-d2589718-5d7f-4981-a975-1dab5fa233e3 None None] MFA Rules not processed for user `89aa67ee7dff4ad5beb433c399dc1b7f`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:55:58.214141 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-d2589718-5d7f-4981-a975-1dab5fa233e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:58.254335 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-9a3b851a-25bd-40e8-9b78-83188068fe94 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:55:58.256853 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-9a3b851a-25bd-40e8-9b78-83188068fe94 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:58.300852 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:55:58.315173 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 980/1973] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:55:58 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:55:58.316205 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2589718-5d7f-4981-a975-1dab5fa233e3 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:58.317717 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-d2589718-5d7f-4981-a975-1dab5fa233e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:58.333231 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:55:58.343341 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 994/1974] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:55:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:55:58.344616 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a3b851a-25bd-40e8-9b78-83188068fe94 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:58.346265 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-9a3b851a-25bd-40e8-9b78-83188068fe94 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:58.456869 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2589718-5d7f-4981-a975-1dab5fa233e3 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:58.467196 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1f4e693-4672-497b-8f4d-12d103e3ad6c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:58.467793 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1f4e693-4672-497b-8f4d-12d103e3ad6c admin admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:58.468120 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1f4e693-4672-497b-8f4d-12d103e3ad6c admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:58.468390 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1f4e693-4672-497b-8f4d-12d103e3ad6c admin admin] PATH_INFO: `/v3/projects` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:58.469062 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1f4e693-4672-497b-8f4d-12d103e3ad6c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:58.472129 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1f4e693-4672-497b-8f4d-12d103e3ad6c admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:58.495526 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a3b851a-25bd-40e8-9b78-83188068fe94 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:58.501818 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbbc1449-41de-4a5c-9125-3ef36912bcbd service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14176b54cf6640abb3b1c14e3824d4f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:58.503941 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbbc1449-41de-4a5c-9125-3ef36912bcbd service glance] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:58.503941 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbbc1449-41de-4a5c-9125-3ef36912bcbd service glance] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:58.503941 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbbc1449-41de-4a5c-9125-3ef36912bcbd service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:58.516013 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbbc1449-41de-4a5c-9125-3ef36912bcbd service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:58.517528 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbbc1449-41de-4a5c-9125-3ef36912bcbd service glance] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:58.525493 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 981/1975] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 00:55:58 2022] POST /identity/v3/projects => generated 383 bytes in 206 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:55:58.525493 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1f4e693-4672-497b-8f4d-12d103e3ad6c admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:58.527284 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-a1f4e693-4672-497b-8f4d-12d103e3ad6c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:58.533778 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 995/1976] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:55:58 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 190 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:55:58.535088 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbbc1449-41de-4a5c-9125-3ef36912bcbd service glance] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:58.548027 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbbc1449-41de-4a5c-9125-3ef36912bcbd service glance] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:58.638571 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e80f9f12-98f9-457d-8b18-5b0aeee291d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:58.639463 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e80f9f12-98f9-457d-8b18-5b0aeee291d4 admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:58.639950 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e80f9f12-98f9-457d-8b18-5b0aeee291d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:58.640390 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e80f9f12-98f9-457d-8b18-5b0aeee291d4 admin admin] PATH_INFO: `/v3/users` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:58.641478 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e80f9f12-98f9-457d-8b18-5b0aeee291d4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:58.644029 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e80f9f12-98f9-457d-8b18-5b0aeee291d4 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:58.685006 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1f4e693-4672-497b-8f4d-12d103e3ad6c admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:58.690551 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70d699b3-9331-4a6e-9f23-f83c2a777df2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:58.691201 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70d699b3-9331-4a6e-9f23-f83c2a777df2 service neutron] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:58.691485 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70d699b3-9331-4a6e-9f23-f83c2a777df2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:58.691761 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70d699b3-9331-4a6e-9f23-f83c2a777df2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:58.695911 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 996/1977] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 00:55:58 2022] POST /identity/v3/users => generated 292 bytes in 162 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:55:58.702020 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e80f9f12-98f9-457d-8b18-5b0aeee291d4 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:58.722319 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70d699b3-9331-4a6e-9f23-f83c2a777df2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:58.723037 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e80f9f12-98f9-457d-8b18-5b0aeee291d4 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:58.724174 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70d699b3-9331-4a6e-9f23-f83c2a777df2 service neutron] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:58.735566 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83746309-9986-4c92-bc45-f469b2e3dc6f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:58.736205 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83746309-9986-4c92-bc45-f469b2e3dc6f admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:58.736635 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83746309-9986-4c92-bc45-f469b2e3dc6f admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:58.736755 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 982/1978] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:55:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 214 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:55:58.736851 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83746309-9986-4c92-bc45-f469b2e3dc6f admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:58.737432 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83746309-9986-4c92-bc45-f469b2e3dc6f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:58.739040 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83746309-9986-4c92-bc45-f469b2e3dc6f admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:58.754159 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 997/1979] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:55:58 2022] GET /identity/v3/roles => generated 1420 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 00:55:58.758415 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70d699b3-9331-4a6e-9f23-f83c2a777df2 service neutron] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:58.769572 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70d699b3-9331-4a6e-9f23-f83c2a777df2 service neutron] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:58.787689 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edef9727-99ca-4804-b7b2-6c19cb5f765d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:58.802439 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edef9727-99ca-4804-b7b2-6c19cb5f765d admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:58.811533 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edef9727-99ca-4804-b7b2-6c19cb5f765d admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:58.816523 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edef9727-99ca-4804-b7b2-6c19cb5f765d admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:58.817539 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edef9727-99ca-4804-b7b2-6c19cb5f765d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:58.819437 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edef9727-99ca-4804-b7b2-6c19cb5f765d admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:58.834255 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 983/1980] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:55:58 2022] GET /identity/v3/roles => generated 1420 bytes in 76 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 00:55:58.841906 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83746309-9986-4c92-bc45-f469b2e3dc6f admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:58.860529 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83746309-9986-4c92-bc45-f469b2e3dc6f admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:58.895925 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce6b2d9e-276c-45dd-85e5-b252a689d4b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:58.896730 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce6b2d9e-276c-45dd-85e5-b252a689d4b5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:58.896882 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce6b2d9e-276c-45dd-85e5-b252a689d4b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:58.897192 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce6b2d9e-276c-45dd-85e5-b252a689d4b5 admin admin] PATH_INFO: `/v3/projects/7f1d082842b84f289eb2e3b876f41d50/users/9f85304aa9414cbe9b7435b873003649/roles/0487ef7e943e4701915cb8ce0f2899d7` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:58.976793 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce6b2d9e-276c-45dd-85e5-b252a689d4b5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7f1d082842b84f289eb2e3b876f41d50, user_id=9f85304aa9414cbe9b7435b873003649, role_id=0487ef7e943e4701915cb8ce0f2899d7)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:58.980323 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce6b2d9e-276c-45dd-85e5-b252a689d4b5 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:59.029925 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 998/1981] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 00:55:58 2022] PUT /identity/v3/projects/7f1d082842b84f289eb2e3b876f41d50/users/9f85304aa9414cbe9b7435b873003649/roles/0487ef7e943e4701915cb8ce0f2899d7 => generated 0 bytes in 182 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:55:59.033406 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a68cf2c-f7ee-4e42-abce-60a3482a1b90 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:59.034451 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a68cf2c-f7ee-4e42-abce-60a3482a1b90 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:59.035170 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a68cf2c-f7ee-4e42-abce-60a3482a1b90 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:59.072506 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-8a68cf2c-f7ee-4e42-abce-60a3482a1b90 None None] MFA Rules not processed for user `9f85304aa9414cbe9b7435b873003649`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:55:59.074283 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-8a68cf2c-f7ee-4e42-abce-60a3482a1b90 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:59.205655 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 984/1982] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:55:59 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:55:59.210982 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce6b2d9e-276c-45dd-85e5-b252a689d4b5 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:59.267983 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:55:59.338326 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce6b2d9e-276c-45dd-85e5-b252a689d4b5 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:59.348735 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-246a00c4-3726-46ef-a1df-02552280186c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:59.348735 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-246a00c4-3726-46ef-a1df-02552280186c admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:59.348735 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-246a00c4-3726-46ef-a1df-02552280186c admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:59.349038 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-246a00c4-3726-46ef-a1df-02552280186c admin admin] PATH_INFO: `/v3/projects` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:59.349728 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-246a00c4-3726-46ef-a1df-02552280186c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:59.351951 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-246a00c4-3726-46ef-a1df-02552280186c admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:59.394762 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 999/1983] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 00:55:59 2022] POST /identity/v3/projects => generated 385 bytes in 185 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:55:59.399107 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a68cf2c-f7ee-4e42-abce-60a3482a1b90 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:59.412311 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a68cf2c-f7ee-4e42-abce-60a3482a1b90 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:59.429470 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:55:59.517313 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3aaf2b7-80ee-4ed7-8528-0f8d4390e633 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:59.518158 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3aaf2b7-80ee-4ed7-8528-0f8d4390e633 admin admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:59.520112 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3aaf2b7-80ee-4ed7-8528-0f8d4390e633 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:59.521852 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3aaf2b7-80ee-4ed7-8528-0f8d4390e633 admin admin] PATH_INFO: `/v3/users` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:59.523892 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3aaf2b7-80ee-4ed7-8528-0f8d4390e633 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:59.528833 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3aaf2b7-80ee-4ed7-8528-0f8d4390e633 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:59.566995 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-246a00c4-3726-46ef-a1df-02552280186c admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:59.569042 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-246a00c4-3726-46ef-a1df-02552280186c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:59.584095 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 985/1984] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 00:55:59 2022] POST /identity/v3/users => generated 299 bytes in 185 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:55:59.588511 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3aaf2b7-80ee-4ed7-8528-0f8d4390e633 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:59.603145 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-246a00c4-3726-46ef-a1df-02552280186c admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:59.609962 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3aaf2b7-80ee-4ed7-8528-0f8d4390e633 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:59.615600 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7805af01-d9da-4177-afb6-772889cadd1f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:59.616303 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7805af01-d9da-4177-afb6-772889cadd1f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:59.616588 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7805af01-d9da-4177-afb6-772889cadd1f admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:59.617694 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7805af01-d9da-4177-afb6-772889cadd1f admin admin] PATH_INFO: `/v3/projects/38f4fef6151241d494acc66a152703f3` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:59.620352 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7805af01-d9da-4177-afb6-772889cadd1f admin admin] RBAC: Authorizing `identity:delete_project(project_id=38f4fef6151241d494acc66a152703f3)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:59.623167 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7805af01-d9da-4177-afb6-772889cadd1f admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:59.628689 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f3a55e7-0727-4e80-9265-f1333067677c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:59.629632 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f3a55e7-0727-4e80-9265-f1333067677c admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:59.630239 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f3a55e7-0727-4e80-9265-f1333067677c admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:59.630610 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f3a55e7-0727-4e80-9265-f1333067677c admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:59.631482 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f3a55e7-0727-4e80-9265-f1333067677c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:59.634774 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f3a55e7-0727-4e80-9265-f1333067677c admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:59.681535 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 986/1985] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:55:59 2022] GET /identity/v3/roles => generated 1420 bytes in 94 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 00:55:59.683215 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f3a55e7-0727-4e80-9265-f1333067677c admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:59.695013 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f3a55e7-0727-4e80-9265-f1333067677c admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:55:59.704971 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-7805af01-d9da-4177-afb6-772889cadd1f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '38f4fef6151241d494acc66a152703f3', 'request_id': 'req-7805af01-d9da-4177-afb6-772889cadd1f'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:59.714332 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-7805af01-d9da-4177-afb6-772889cadd1f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '38f4fef6151241d494acc66a152703f3', 'request_id': 'req-7805af01-d9da-4177-afb6-772889cadd1f'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:59.727915 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-7805af01-d9da-4177-afb6-772889cadd1f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '38f4fef6151241d494acc66a152703f3', 'request_id': 'req-7805af01-d9da-4177-afb6-772889cadd1f'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:59.744319 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-7805af01-d9da-4177-afb6-772889cadd1f admin admin] The token cache is being invalidate because project 38f4fef6151241d494acc66a152703f3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81508) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 00:55:59.744652 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-7805af01-d9da-4177-afb6-772889cadd1f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:59.745432 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-7805af01-d9da-4177-afb6-772889cadd1f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:55:59.747263 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1000/1986] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 00:55:59 2022] DELETE /identity/v3/projects/38f4fef6151241d494acc66a152703f3 => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 28 00:55:59.752500 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b379de74-d042-4041-8f53-50e9c56b3df8 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:59.752786 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b379de74-d042-4041-8f53-50e9c56b3df8 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:59.753332 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b379de74-d042-4041-8f53-50e9c56b3df8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:59.790952 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-b379de74-d042-4041-8f53-50e9c56b3df8 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:55:59.791618 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d773c1c-01fc-4148-b69b-2cfaa80d4552 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:55:59.792501 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-b379de74-d042-4041-8f53-50e9c56b3df8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:59.792749 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d773c1c-01fc-4148-b69b-2cfaa80d4552 admin admin] REQUEST_METHOD: `PUT` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:59.793455 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d773c1c-01fc-4148-b69b-2cfaa80d4552 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:59.794059 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d773c1c-01fc-4148-b69b-2cfaa80d4552 admin admin] PATH_INFO: `/v3/projects/9fdffeebe5f549d296f200d564f2652b/users/be21f3422e4f4abfb1fa71814fedd4bd/roles/f08d6043261747e3b4763c744eef7dfb` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:59.803696 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1001/1987] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:55:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 51 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:55:59.810792 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6c66dfc-f669-49a2-81f2-b0ac0480be9e None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:55:59.811005 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6c66dfc-f669-49a2-81f2-b0ac0480be9e None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:55:59.811290 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6c66dfc-f669-49a2-81f2-b0ac0480be9e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:55:59.828206 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d773c1c-01fc-4148-b69b-2cfaa80d4552 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9fdffeebe5f549d296f200d564f2652b, user_id=be21f3422e4f4abfb1fa71814fedd4bd, role_id=f08d6043261747e3b4763c744eef7dfb)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:55:59.831834 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d773c1c-01fc-4148-b69b-2cfaa80d4552 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:55:59.855823 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 987/1988] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 00:55:59 2022] PUT /identity/v3/projects/9fdffeebe5f549d296f200d564f2652b/users/be21f3422e4f4abfb1fa71814fedd4bd/roles/f08d6043261747e3b4763c744eef7dfb => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:55:59.861026 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d773c1c-01fc-4148-b69b-2cfaa80d4552 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:55:59.866497 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-7d773c1c-01fc-4148-b69b-2cfaa80d4552 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:55:59.870233 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-e6c66dfc-f669-49a2-81f2-b0ac0480be9e None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:55:59.872299 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-e6c66dfc-f669-49a2-81f2-b0ac0480be9e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:56:00.048149 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d773c1c-01fc-4148-b69b-2cfaa80d4552 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:00.071416 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1002/1989] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:55:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 259 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 28 00:56:00.072822 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6c66dfc-f669-49a2-81f2-b0ac0480be9e None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:00.081295 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2389d738-f0d6-4d10-bc8a-f600eceb1f45 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:00.082279 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2389d738-f0d6-4d10-bc8a-f600eceb1f45 admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:00.082556 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2389d738-f0d6-4d10-bc8a-f600eceb1f45 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:00.082827 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2389d738-f0d6-4d10-bc8a-f600eceb1f45 admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:00.083032 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6c66dfc-f669-49a2-81f2-b0ac0480be9e None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:00.083480 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2389d738-f0d6-4d10-bc8a-f600eceb1f45 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:00.085124 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2389d738-f0d6-4d10-bc8a-f600eceb1f45 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:00.089280 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c0b3d8e-d6fa-4761-96f7-f6502146170c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:00.095045 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c0b3d8e-d6fa-4761-96f7-f6502146170c admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:00.095045 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c0b3d8e-d6fa-4761-96f7-f6502146170c admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:00.095045 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c0b3d8e-d6fa-4761-96f7-f6502146170c admin admin] PATH_INFO: `/v3/domains` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:00.095045 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c0b3d8e-d6fa-4761-96f7-f6502146170c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:56:00.095045 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c0b3d8e-d6fa-4761-96f7-f6502146170c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:00.096375 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c0b3d8e-d6fa-4761-96f7-f6502146170c admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:00.107599 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 988/1990] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:55:59 2022] GET /identity/v3/roles => generated 1420 bytes in 247 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 00:56:00.113253 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1003/1991] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Aug 28 00:56:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 28 00:56:00.118365 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2389d738-f0d6-4d10-bc8a-f600eceb1f45 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:00.121363 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c0b3d8e-d6fa-4761-96f7-f6502146170c admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:00.139740 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c0b3d8e-d6fa-4761-96f7-f6502146170c admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:00.143635 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2389d738-f0d6-4d10-bc8a-f600eceb1f45 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:00.145485 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3519da6e-c472-4a26-9127-00d4a1cefc5f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:00.146304 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3519da6e-c472-4a26-9127-00d4a1cefc5f admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:00.146744 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3519da6e-c472-4a26-9127-00d4a1cefc5f admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:00.147024 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3519da6e-c472-4a26-9127-00d4a1cefc5f admin admin] PATH_INFO: `/v3/projects` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:00.147732 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3519da6e-c472-4a26-9127-00d4a1cefc5f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:00.149887 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3519da6e-c472-4a26-9127-00d4a1cefc5f admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:00.150170 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e485a576-6314-4d72-a150-c385b32a7bd6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:00.150899 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e485a576-6314-4d72-a150-c385b32a7bd6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:00.151844 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e485a576-6314-4d72-a150-c385b32a7bd6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:00.152123 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e485a576-6314-4d72-a150-c385b32a7bd6 admin admin] PATH_INFO: `/v3/projects/9fdffeebe5f549d296f200d564f2652b/users/be21f3422e4f4abfb1fa71814fedd4bd/roles/f08d6043261747e3b4763c744eef7dfb` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:00.154540 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e485a576-6314-4d72-a150-c385b32a7bd6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9fdffeebe5f549d296f200d564f2652b, user_id=be21f3422e4f4abfb1fa71814fedd4bd, role_id=f08d6043261747e3b4763c744eef7dfb)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:00.155606 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:56:00.158148 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e485a576-6314-4d72-a150-c385b32a7bd6 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:00.173984 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:56:00.190189 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 989/1992] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 00:56:00 2022] PUT /identity/v3/projects/9fdffeebe5f549d296f200d564f2652b/users/be21f3422e4f4abfb1fa71814fedd4bd/roles/f08d6043261747e3b4763c744eef7dfb => generated 0 bytes in 72 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:56:00.202349 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c74a9234-40bb-4cd4-9b37-3767f982a9f0 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:00.202712 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c74a9234-40bb-4cd4-9b37-3767f982a9f0 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:00.202916 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c74a9234-40bb-4cd4-9b37-3767f982a9f0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:00.208764 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1004/1993] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 00:56:00 2022] POST /identity/v3/projects => generated 375 bytes in 86 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:56:00.212787 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3519da6e-c472-4a26-9127-00d4a1cefc5f admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:00.253333 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-c74a9234-40bb-4cd4-9b37-3767f982a9f0 None None] MFA Rules not processed for user `be21f3422e4f4abfb1fa71814fedd4bd`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:56:00.254629 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-c74a9234-40bb-4cd4-9b37-3767f982a9f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:56:00.313092 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3519da6e-c472-4a26-9127-00d4a1cefc5f admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:00.321671 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8ead7c2-4c80-441d-8a60-f7cad14d7231 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:00.322396 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8ead7c2-4c80-441d-8a60-f7cad14d7231 admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:00.322595 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8ead7c2-4c80-441d-8a60-f7cad14d7231 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:00.322871 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8ead7c2-4c80-441d-8a60-f7cad14d7231 admin admin] PATH_INFO: `/v3/users` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:00.323513 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8ead7c2-4c80-441d-8a60-f7cad14d7231 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:00.325880 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8ead7c2-4c80-441d-8a60-f7cad14d7231 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:00.381904 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1005/1994] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 00:56:00 2022] POST /identity/v3/users => generated 288 bytes in 170 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:56:00.386083 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8ead7c2-4c80-441d-8a60-f7cad14d7231 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:00.423173 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8ead7c2-4c80-441d-8a60-f7cad14d7231 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:00.437917 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbf00744-ee62-4058-a809-ffffbf772b1b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:00.438607 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbf00744-ee62-4058-a809-ffffbf772b1b admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:00.438837 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbf00744-ee62-4058-a809-ffffbf772b1b admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:00.439115 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbf00744-ee62-4058-a809-ffffbf772b1b admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:00.439782 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbf00744-ee62-4058-a809-ffffbf772b1b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:00.441774 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbf00744-ee62-4058-a809-ffffbf772b1b admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:00.444709 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 990/1995] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:56:00 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 244 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:56:00.446847 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c74a9234-40bb-4cd4-9b37-3767f982a9f0 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:00.450331 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-c74a9234-40bb-4cd4-9b37-3767f982a9f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:56:00.473389 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1006/1996] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:56:00 2022] GET /identity/v3/roles => generated 1420 bytes in 88 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 28 00:56:00.477286 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbf00744-ee62-4058-a809-ffffbf772b1b admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:00.496210 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbf00744-ee62-4058-a809-ffffbf772b1b admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:00.511369 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffc540fd-120f-4b86-81d4-218db99f42bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:00.512659 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffc540fd-120f-4b86-81d4-218db99f42bd admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:00.512659 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffc540fd-120f-4b86-81d4-218db99f42bd admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:00.512659 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffc540fd-120f-4b86-81d4-218db99f42bd admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:00.513224 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffc540fd-120f-4b86-81d4-218db99f42bd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:00.515114 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffc540fd-120f-4b86-81d4-218db99f42bd admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:00.540066 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1007/1997] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:56:00 2022] GET /identity/v3/roles => generated 1420 bytes in 63 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 28 00:56:00.544185 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffc540fd-120f-4b86-81d4-218db99f42bd admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:00.554151 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c74a9234-40bb-4cd4-9b37-3767f982a9f0 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:00.575287 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abfa9f9b-8a5f-4f7b-9786-94c1f29dc4ed service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:00.575870 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abfa9f9b-8a5f-4f7b-9786-94c1f29dc4ed service cinder] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:00.581518 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffc540fd-120f-4b86-81d4-218db99f42bd admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:00.581755 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abfa9f9b-8a5f-4f7b-9786-94c1f29dc4ed service cinder] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:00.581755 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abfa9f9b-8a5f-4f7b-9786-94c1f29dc4ed service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:00.586333 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-abfa9f9b-8a5f-4f7b-9786-94c1f29dc4ed service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:56:00.599409 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-904754ce-1115-4810-9877-cc2df780fbf3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:00.599849 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-904754ce-1115-4810-9877-cc2df780fbf3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:00.600234 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-904754ce-1115-4810-9877-cc2df780fbf3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:00.600524 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-904754ce-1115-4810-9877-cc2df780fbf3 admin admin] PATH_INFO: `/v3/projects/d53428c669da4434b39e90777cb84ac2/users/943e26b5ba5d489e8486a0984962b274/roles/0487ef7e943e4701915cb8ce0f2899d7` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:00.620985 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-904754ce-1115-4810-9877-cc2df780fbf3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d53428c669da4434b39e90777cb84ac2, user_id=943e26b5ba5d489e8486a0984962b274, role_id=0487ef7e943e4701915cb8ce0f2899d7)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:00.625509 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-904754ce-1115-4810-9877-cc2df780fbf3 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:00.658256 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1008/1998] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 00:56:00 2022] PUT /identity/v3/projects/d53428c669da4434b39e90777cb84ac2/users/943e26b5ba5d489e8486a0984962b274/roles/0487ef7e943e4701915cb8ce0f2899d7 => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:56:00.658256 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec902d3-7a27-4d3e-8776-0af036b446bd None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:00.658256 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec902d3-7a27-4d3e-8776-0af036b446bd None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:00.658256 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec902d3-7a27-4d3e-8776-0af036b446bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:00.711896 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-7ec902d3-7a27-4d3e-8776-0af036b446bd None None] MFA Rules not processed for user `943e26b5ba5d489e8486a0984962b274`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:56:00.713493 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-7ec902d3-7a27-4d3e-8776-0af036b446bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:56:00.713685 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abfa9f9b-8a5f-4f7b-9786-94c1f29dc4ed service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:00.715162 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abfa9f9b-8a5f-4f7b-9786-94c1f29dc4ed service cinder] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:00.728421 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 991/1999] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:56:00 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 283 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:56:00.797443 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1009/2000] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:56:00 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:56:00.801896 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abfa9f9b-8a5f-4f7b-9786-94c1f29dc4ed service cinder] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:00.814088 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abfa9f9b-8a5f-4f7b-9786-94c1f29dc4ed service cinder] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:00.861436 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:56:00.915353 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8ef5f1b-1d99-4447-9d6f-5b4a64c91afd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:00.918005 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8ef5f1b-1d99-4447-9d6f-5b4a64c91afd admin admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:00.918446 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8ef5f1b-1d99-4447-9d6f-5b4a64c91afd admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:00.919054 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8ef5f1b-1d99-4447-9d6f-5b4a64c91afd admin admin] PATH_INFO: `/v3/projects` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:00.920514 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8ef5f1b-1d99-4447-9d6f-5b4a64c91afd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:00.925084 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8ef5f1b-1d99-4447-9d6f-5b4a64c91afd admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:00.982945 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 992/2001] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 00:56:00 2022] POST /identity/v3/projects => generated 375 bytes in 181 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 28 00:56:00.987103 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ec902d3-7a27-4d3e-8776-0af036b446bd None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:01.004795 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ec902d3-7a27-4d3e-8776-0af036b446bd None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:01.032906 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:56:01.135018 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6621ded3-3797-4da9-a29a-b5dfb69c3afa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:01.137042 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6621ded3-3797-4da9-a29a-b5dfb69c3afa admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:01.137341 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6621ded3-3797-4da9-a29a-b5dfb69c3afa admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:01.137635 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6621ded3-3797-4da9-a29a-b5dfb69c3afa admin admin] PATH_INFO: `/v3/users` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:01.138368 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6621ded3-3797-4da9-a29a-b5dfb69c3afa admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:01.145119 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6621ded3-3797-4da9-a29a-b5dfb69c3afa admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:01.219340 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8ef5f1b-1d99-4447-9d6f-5b4a64c91afd admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:01.230994 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1010/2002] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 00:56:00 2022] POST /identity/v3/users => generated 294 bytes in 244 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 28 00:56:01.237136 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8ef5f1b-1d99-4447-9d6f-5b4a64c91afd admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:01.247327 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-894d8a3c-49a6-48d1-87e8-b7c2f80ee454 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:01.247974 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-894d8a3c-49a6-48d1-87e8-b7c2f80ee454 admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:01.248520 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-894d8a3c-49a6-48d1-87e8-b7c2f80ee454 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:01.248792 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-894d8a3c-49a6-48d1-87e8-b7c2f80ee454 admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:01.252065 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-894d8a3c-49a6-48d1-87e8-b7c2f80ee454 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:01.257135 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-894d8a3c-49a6-48d1-87e8-b7c2f80ee454 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:01.284508 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6621ded3-3797-4da9-a29a-b5dfb69c3afa admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:01.292148 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 993/2003] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:56:01 2022] GET /identity/v3/roles => generated 1420 bytes in 72 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 28 00:56:01.298895 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6621ded3-3797-4da9-a29a-b5dfb69c3afa admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:01.306350 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15e18618-8b99-405e-a89c-673d7b97b535 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:01.307174 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e18618-8b99-405e-a89c-673d7b97b535 admin admin] REQUEST_METHOD: `PUT` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:01.308135 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e18618-8b99-405e-a89c-673d7b97b535 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:01.308486 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e18618-8b99-405e-a89c-673d7b97b535 admin admin] PATH_INFO: `/v3/projects/af1895b43ecb4bca96a56444d30e272e/users/eed5fff5fc9b45639653d3e70ba6b31b/roles/f08d6043261747e3b4763c744eef7dfb` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:01.332552 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15e18618-8b99-405e-a89c-673d7b97b535 admin admin] RBAC: Authorizing `identity:create_grant(project_id=af1895b43ecb4bca96a56444d30e272e, user_id=eed5fff5fc9b45639653d3e70ba6b31b, role_id=f08d6043261747e3b4763c744eef7dfb)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:01.335704 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15e18618-8b99-405e-a89c-673d7b97b535 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:01.352497 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1011/2004] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 00:56:01 2022] PUT /identity/v3/projects/af1895b43ecb4bca96a56444d30e272e/users/eed5fff5fc9b45639653d3e70ba6b31b/roles/f08d6043261747e3b4763c744eef7dfb => generated 0 bytes in 69 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:56:01.356579 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-894d8a3c-49a6-48d1-87e8-b7c2f80ee454 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:01.367408 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-894d8a3c-49a6-48d1-87e8-b7c2f80ee454 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:01.457602 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d1673cb-c75f-442f-9942-95c95318a81b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:01.458232 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d1673cb-c75f-442f-9942-95c95318a81b admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:01.458518 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d1673cb-c75f-442f-9942-95c95318a81b admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:01.458794 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d1673cb-c75f-442f-9942-95c95318a81b admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:01.459501 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d1673cb-c75f-442f-9942-95c95318a81b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:01.461152 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d1673cb-c75f-442f-9942-95c95318a81b admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:01.473470 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 994/2005] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:56:01 2022] GET /identity/v3/roles => generated 1420 bytes in 117 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 00:56:01.478376 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15e18618-8b99-405e-a89c-673d7b97b535 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:01.489685 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15e18618-8b99-405e-a89c-673d7b97b535 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:01.495084 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b081a1fe-7582-4296-9260-26cb877601ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:01.495768 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b081a1fe-7582-4296-9260-26cb877601ac admin admin] REQUEST_METHOD: `PUT` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:01.496049 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b081a1fe-7582-4296-9260-26cb877601ac admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:01.496325 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b081a1fe-7582-4296-9260-26cb877601ac admin admin] PATH_INFO: `/v3/projects/af1895b43ecb4bca96a56444d30e272e/users/eed5fff5fc9b45639653d3e70ba6b31b/roles/f08d6043261747e3b4763c744eef7dfb` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:01.498324 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b081a1fe-7582-4296-9260-26cb877601ac admin admin] RBAC: Authorizing `identity:create_grant(project_id=af1895b43ecb4bca96a56444d30e272e, user_id=eed5fff5fc9b45639653d3e70ba6b31b, role_id=f08d6043261747e3b4763c744eef7dfb)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:01.501187 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b081a1fe-7582-4296-9260-26cb877601ac admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:01.514259 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1012/2006] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 00:56:01 2022] PUT /identity/v3/projects/af1895b43ecb4bca96a56444d30e272e/users/eed5fff5fc9b45639653d3e70ba6b31b/roles/f08d6043261747e3b4763c744eef7dfb => generated 0 bytes in 36 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:56:01.524900 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-507f4e0b-ebd7-4daa-8aaa-cba93d1a8385 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:01.525217 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-507f4e0b-ebd7-4daa-8aaa-cba93d1a8385 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:01.525518 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-507f4e0b-ebd7-4daa-8aaa-cba93d1a8385 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:01.561649 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-507f4e0b-ebd7-4daa-8aaa-cba93d1a8385 None None] MFA Rules not processed for user `eed5fff5fc9b45639653d3e70ba6b31b`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:56:01.563144 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-507f4e0b-ebd7-4daa-8aaa-cba93d1a8385 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:56:01.647848 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 995/2007] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:56:01 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:56:01.659481 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b081a1fe-7582-4296-9260-26cb877601ac admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:01.743938 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b081a1fe-7582-4296-9260-26cb877601ac admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:01.748812 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cab8ab51-297d-4076-bb98-a43273f78bfb service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:01.749649 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cab8ab51-297d-4076-bb98-a43273f78bfb service cinder] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:01.750004 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cab8ab51-297d-4076-bb98-a43273f78bfb service cinder] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:01.750343 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cab8ab51-297d-4076-bb98-a43273f78bfb service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:01.850234 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cab8ab51-297d-4076-bb98-a43273f78bfb service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:01.851780 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cab8ab51-297d-4076-bb98-a43273f78bfb service cinder] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:01.873092 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1013/2008] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:56:01 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 208 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:56:02.592810 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-507f4e0b-ebd7-4daa-8aaa-cba93d1a8385 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:02.594289 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-507f4e0b-ebd7-4daa-8aaa-cba93d1a8385 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:56:02.606643 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-507f4e0b-ebd7-4daa-8aaa-cba93d1a8385 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:02.620911 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b361c8f4-1f0a-42c6-9d58-27eb46a63d2c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:02.621508 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b361c8f4-1f0a-42c6-9d58-27eb46a63d2c service cinder] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:02.621854 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b361c8f4-1f0a-42c6-9d58-27eb46a63d2c service cinder] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:02.622808 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b361c8f4-1f0a-42c6-9d58-27eb46a63d2c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:02.625103 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-b361c8f4-1f0a-42c6-9d58-27eb46a63d2c service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:56:02.667560 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:56:02.713318 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b361c8f4-1f0a-42c6-9d58-27eb46a63d2c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:02.716623 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b361c8f4-1f0a-42c6-9d58-27eb46a63d2c service cinder] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:02.733380 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 996/2009] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:56:02 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 141 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:56:03.497915 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cab8ab51-297d-4076-bb98-a43273f78bfb service cinder] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:03.515183 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cab8ab51-297d-4076-bb98-a43273f78bfb service cinder] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:03.519071 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46389af4-908e-48b1-8363-c9a4a50d404f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:03.519701 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46389af4-908e-48b1-8363-c9a4a50d404f service cinder] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:03.520014 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46389af4-908e-48b1-8363-c9a4a50d404f service cinder] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:03.520314 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46389af4-908e-48b1-8363-c9a4a50d404f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:03.555589 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:56:03.611827 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46389af4-908e-48b1-8363-c9a4a50d404f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:03.613701 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46389af4-908e-48b1-8363-c9a4a50d404f service cinder] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:03.629726 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1014/2010] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:56:03 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 132 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 28 00:56:04.793009 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b361c8f4-1f0a-42c6-9d58-27eb46a63d2c service cinder] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:04.877273 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b361c8f4-1f0a-42c6-9d58-27eb46a63d2c service cinder] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:04.882525 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a24e6e2c-7c41-42a6-b259-4f18398b1a4b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:04.883242 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a24e6e2c-7c41-42a6-b259-4f18398b1a4b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:04.883538 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a24e6e2c-7c41-42a6-b259-4f18398b1a4b admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:04.883936 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a24e6e2c-7c41-42a6-b259-4f18398b1a4b admin admin] PATH_INFO: `/v3/users/943e26b5ba5d489e8486a0984962b274` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:04.885424 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a24e6e2c-7c41-42a6-b259-4f18398b1a4b admin admin] RBAC: Authorizing `identity:delete_user(user_id=943e26b5ba5d489e8486a0984962b274)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:04.887427 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a24e6e2c-7c41-42a6-b259-4f18398b1a4b admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:04.941944 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-a24e6e2c-7c41-42a6-b259-4f18398b1a4b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '943e26b5ba5d489e8486a0984962b274'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:04.957776 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-a24e6e2c-7c41-42a6-b259-4f18398b1a4b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '943e26b5ba5d489e8486a0984962b274'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:04.969209 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-a24e6e2c-7c41-42a6-b259-4f18398b1a4b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '943e26b5ba5d489e8486a0984962b274'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:04.977538 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-a24e6e2c-7c41-42a6-b259-4f18398b1a4b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '943e26b5ba5d489e8486a0984962b274'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:04.978249 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-a24e6e2c-7c41-42a6-b259-4f18398b1a4b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '943e26b5ba5d489e8486a0984962b274'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:04.988001 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 997/2011] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 00:56:04 2022] DELETE /identity/v3/users/943e26b5ba5d489e8486a0984962b274 => generated 0 bytes in 193 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:56:04.989775 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55889d43-3231-489e-9ab9-f43b5087f9a9 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:04.990336 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55889d43-3231-489e-9ab9-f43b5087f9a9 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:04.990336 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55889d43-3231-489e-9ab9-f43b5087f9a9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:05.013566 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-55889d43-3231-489e-9ab9-f43b5087f9a9 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:56:05.014859 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-55889d43-3231-489e-9ab9-f43b5087f9a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:56:05.113539 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1015/2012] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:56:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:56:05.125656 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a24e6e2c-7c41-42a6-b259-4f18398b1a4b admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:05.127058 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-a24e6e2c-7c41-42a6-b259-4f18398b1a4b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:56:05.190240 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:56:05.194978 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a24e6e2c-7c41-42a6-b259-4f18398b1a4b admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:05.199926 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1769f950-2504-4764-85a8-b6ec43ba1a3d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:05.200546 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1769f950-2504-4764-85a8-b6ec43ba1a3d service neutron] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:05.201368 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1769f950-2504-4764-85a8-b6ec43ba1a3d service neutron] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:05.201642 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1769f950-2504-4764-85a8-b6ec43ba1a3d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:05.212489 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1769f950-2504-4764-85a8-b6ec43ba1a3d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:05.214086 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1769f950-2504-4764-85a8-b6ec43ba1a3d service neutron] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:05.224832 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 998/2013] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:56:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:56:05.652117 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55889d43-3231-489e-9ab9-f43b5087f9a9 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:05.653699 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-55889d43-3231-489e-9ab9-f43b5087f9a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:56:05.667145 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55889d43-3231-489e-9ab9-f43b5087f9a9 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:05.672548 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ba92a62-e3ea-4b3a-8fa1-063a2bceb77d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:05.673451 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ba92a62-e3ea-4b3a-8fa1-063a2bceb77d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:05.673986 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ba92a62-e3ea-4b3a-8fa1-063a2bceb77d admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:05.674535 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ba92a62-e3ea-4b3a-8fa1-063a2bceb77d admin admin] PATH_INFO: `/v3/projects/d53428c669da4434b39e90777cb84ac2` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:05.676985 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ba92a62-e3ea-4b3a-8fa1-063a2bceb77d admin admin] RBAC: Authorizing `identity:delete_project(project_id=d53428c669da4434b39e90777cb84ac2)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:05.681803 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ba92a62-e3ea-4b3a-8fa1-063a2bceb77d admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:05.742845 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-6ba92a62-e3ea-4b3a-8fa1-063a2bceb77d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd53428c669da4434b39e90777cb84ac2', 'request_id': 'req-6ba92a62-e3ea-4b3a-8fa1-063a2bceb77d'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:05.749295 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-6ba92a62-e3ea-4b3a-8fa1-063a2bceb77d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd53428c669da4434b39e90777cb84ac2', 'request_id': 'req-6ba92a62-e3ea-4b3a-8fa1-063a2bceb77d'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:05.767858 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-6ba92a62-e3ea-4b3a-8fa1-063a2bceb77d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd53428c669da4434b39e90777cb84ac2', 'request_id': 'req-6ba92a62-e3ea-4b3a-8fa1-063a2bceb77d'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:05.785830 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-6ba92a62-e3ea-4b3a-8fa1-063a2bceb77d admin admin] The token cache is being invalidate because project d53428c669da4434b39e90777cb84ac2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81508) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 00:56:05.786439 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-6ba92a62-e3ea-4b3a-8fa1-063a2bceb77d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:05.787180 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-6ba92a62-e3ea-4b3a-8fa1-063a2bceb77d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:05.789021 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1016/2014] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 00:56:05 2022] DELETE /identity/v3/projects/d53428c669da4434b39e90777cb84ac2 => generated 0 bytes in 137 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:56:05.793005 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1769f950-2504-4764-85a8-b6ec43ba1a3d service neutron] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:05.804068 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1769f950-2504-4764-85a8-b6ec43ba1a3d service neutron] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:05.875415 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b4364e6-5027-4bf7-8d51-354bc8f502d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:05.876074 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b4364e6-5027-4bf7-8d51-354bc8f502d9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:05.876413 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b4364e6-5027-4bf7-8d51-354bc8f502d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:05.876693 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b4364e6-5027-4bf7-8d51-354bc8f502d9 admin admin] PATH_INFO: `/v3/users/eed5fff5fc9b45639653d3e70ba6b31b` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:05.878032 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b4364e6-5027-4bf7-8d51-354bc8f502d9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=eed5fff5fc9b45639653d3e70ba6b31b)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:05.880163 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b4364e6-5027-4bf7-8d51-354bc8f502d9 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:05.952085 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-5b4364e6-5027-4bf7-8d51-354bc8f502d9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'eed5fff5fc9b45639653d3e70ba6b31b'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:05.974260 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-5b4364e6-5027-4bf7-8d51-354bc8f502d9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'eed5fff5fc9b45639653d3e70ba6b31b'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:05.982718 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-5b4364e6-5027-4bf7-8d51-354bc8f502d9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'eed5fff5fc9b45639653d3e70ba6b31b'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:05.994751 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-5b4364e6-5027-4bf7-8d51-354bc8f502d9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'eed5fff5fc9b45639653d3e70ba6b31b'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:05.995882 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-5b4364e6-5027-4bf7-8d51-354bc8f502d9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'eed5fff5fc9b45639653d3e70ba6b31b'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:06.004641 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 999/2015] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 00:56:05 2022] DELETE /identity/v3/users/eed5fff5fc9b45639653d3e70ba6b31b => generated 0 bytes in 212 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 28 00:56:06.351334 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ba92a62-e3ea-4b3a-8fa1-063a2bceb77d admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:06.353194 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-6ba92a62-e3ea-4b3a-8fa1-063a2bceb77d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:56:06.374811 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ba92a62-e3ea-4b3a-8fa1-063a2bceb77d admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:06.427442 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:56:06.451354 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ac82b42-a2ae-448b-b7aa-c2d5de7b1c1a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:06.452093 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ac82b42-a2ae-448b-b7aa-c2d5de7b1c1a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:06.452395 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ac82b42-a2ae-448b-b7aa-c2d5de7b1c1a admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:06.452670 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ac82b42-a2ae-448b-b7aa-c2d5de7b1c1a admin admin] PATH_INFO: `/v3/projects/af1895b43ecb4bca96a56444d30e272e` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:06.453972 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ac82b42-a2ae-448b-b7aa-c2d5de7b1c1a admin admin] RBAC: Authorizing `identity:delete_project(project_id=af1895b43ecb4bca96a56444d30e272e)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:06.455934 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ac82b42-a2ae-448b-b7aa-c2d5de7b1c1a admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:06.514052 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-3ac82b42-a2ae-448b-b7aa-c2d5de7b1c1a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'af1895b43ecb4bca96a56444d30e272e', 'request_id': 'req-3ac82b42-a2ae-448b-b7aa-c2d5de7b1c1a'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:06.525444 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-3ac82b42-a2ae-448b-b7aa-c2d5de7b1c1a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'af1895b43ecb4bca96a56444d30e272e', 'request_id': 'req-3ac82b42-a2ae-448b-b7aa-c2d5de7b1c1a'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:06.541230 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-3ac82b42-a2ae-448b-b7aa-c2d5de7b1c1a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'af1895b43ecb4bca96a56444d30e272e', 'request_id': 'req-3ac82b42-a2ae-448b-b7aa-c2d5de7b1c1a'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:06.558086 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-3ac82b42-a2ae-448b-b7aa-c2d5de7b1c1a admin admin] The token cache is being invalidate because project af1895b43ecb4bca96a56444d30e272e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81508) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 00:56:06.558387 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-3ac82b42-a2ae-448b-b7aa-c2d5de7b1c1a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:06.559143 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-3ac82b42-a2ae-448b-b7aa-c2d5de7b1c1a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:06.560630 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1017/2016] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 00:56:06 2022] DELETE /identity/v3/projects/af1895b43ecb4bca96a56444d30e272e => generated 0 bytes in 210 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:56:08.679749 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b4364e6-5027-4bf7-8d51-354bc8f502d9 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:08.681355 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-5b4364e6-5027-4bf7-8d51-354bc8f502d9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:56:08.694358 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b4364e6-5027-4bf7-8d51-354bc8f502d9 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:08.759070 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6417e57-e594-406a-8b8b-be549db59fe6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:08.759713 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6417e57-e594-406a-8b8b-be549db59fe6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:08.759993 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6417e57-e594-406a-8b8b-be549db59fe6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:08.760350 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6417e57-e594-406a-8b8b-be549db59fe6 admin admin] PATH_INFO: `/v3/users/f8f56a6153f841de8612b40f65d8db69` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:08.761660 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6417e57-e594-406a-8b8b-be549db59fe6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f8f56a6153f841de8612b40f65d8db69)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:08.763742 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6417e57-e594-406a-8b8b-be549db59fe6 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:08.778827 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:56:08.821215 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-d6417e57-e594-406a-8b8b-be549db59fe6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f8f56a6153f841de8612b40f65d8db69'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:08.834844 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-d6417e57-e594-406a-8b8b-be549db59fe6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f8f56a6153f841de8612b40f65d8db69'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:08.843756 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-d6417e57-e594-406a-8b8b-be549db59fe6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f8f56a6153f841de8612b40f65d8db69'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:08.852044 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-d6417e57-e594-406a-8b8b-be549db59fe6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f8f56a6153f841de8612b40f65d8db69'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:08.852785 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-d6417e57-e594-406a-8b8b-be549db59fe6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f8f56a6153f841de8612b40f65d8db69'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:08.859138 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1000/2017] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 00:56:08 2022] DELETE /identity/v3/users/f8f56a6153f841de8612b40f65d8db69 => generated 0 bytes in 180 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:56:08.863934 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b4f8310-3925-47c4-86b6-9f269eefdcca None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:08.864467 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b4f8310-3925-47c4-86b6-9f269eefdcca None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:08.864585 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b4f8310-3925-47c4-86b6-9f269eefdcca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:08.899038 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-5b4f8310-3925-47c4-86b6-9f269eefdcca None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:56:08.900523 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-5b4f8310-3925-47c4-86b6-9f269eefdcca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:56:08.970980 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1018/2018] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:56:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:56:08.982350 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6417e57-e594-406a-8b8b-be549db59fe6 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:08.984224 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-d6417e57-e594-406a-8b8b-be549db59fe6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:56:09.053651 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6417e57-e594-406a-8b8b-be549db59fe6 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:09.058919 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7edf23e0-7042-48dd-8570-d7231b72700b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:09.059536 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7edf23e0-7042-48dd-8570-d7231b72700b service neutron] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:09.059806 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7edf23e0-7042-48dd-8570-d7231b72700b service neutron] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:09.060071 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7edf23e0-7042-48dd-8570-d7231b72700b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:09.070718 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7edf23e0-7042-48dd-8570-d7231b72700b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:09.072203 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7edf23e0-7042-48dd-8570-d7231b72700b service neutron] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:09.082976 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1001/2019] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:56:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:56:09.375062 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b4f8310-3925-47c4-86b6-9f269eefdcca None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:09.376811 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-5b4f8310-3925-47c4-86b6-9f269eefdcca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:56:09.391720 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b4f8310-3925-47c4-86b6-9f269eefdcca None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:09.397071 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-043846df-fa4d-4556-abf1-02cc6dcde127 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:09.397799 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-043846df-fa4d-4556-abf1-02cc6dcde127 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:09.398325 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-043846df-fa4d-4556-abf1-02cc6dcde127 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:09.398673 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-043846df-fa4d-4556-abf1-02cc6dcde127 admin admin] PATH_INFO: `/v3/projects/6e52a2d256554d84b23f97c15bcc06a4` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:09.400043 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-043846df-fa4d-4556-abf1-02cc6dcde127 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6e52a2d256554d84b23f97c15bcc06a4)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:09.402278 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-043846df-fa4d-4556-abf1-02cc6dcde127 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:09.494154 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-043846df-fa4d-4556-abf1-02cc6dcde127 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6e52a2d256554d84b23f97c15bcc06a4', 'request_id': 'req-043846df-fa4d-4556-abf1-02cc6dcde127'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:09.503607 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-043846df-fa4d-4556-abf1-02cc6dcde127 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6e52a2d256554d84b23f97c15bcc06a4', 'request_id': 'req-043846df-fa4d-4556-abf1-02cc6dcde127'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:09.510160 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-043846df-fa4d-4556-abf1-02cc6dcde127 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6e52a2d256554d84b23f97c15bcc06a4', 'request_id': 'req-043846df-fa4d-4556-abf1-02cc6dcde127'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:09.515990 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:56:09.527479 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-043846df-fa4d-4556-abf1-02cc6dcde127 admin admin] The token cache is being invalidate because project 6e52a2d256554d84b23f97c15bcc06a4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81508) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 00:56:09.527777 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-043846df-fa4d-4556-abf1-02cc6dcde127 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:09.528549 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-043846df-fa4d-4556-abf1-02cc6dcde127 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:09.530373 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1019/2020] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 00:56:09 2022] DELETE /identity/v3/projects/6e52a2d256554d84b23f97c15bcc06a4 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:56:09.534224 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7edf23e0-7042-48dd-8570-d7231b72700b service neutron] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:09.546725 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7edf23e0-7042-48dd-8570-d7231b72700b service neutron] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:09.610102 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-951b2844-2765-4954-826b-dec426f39e86 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:09.610779 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-951b2844-2765-4954-826b-dec426f39e86 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:09.611057 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-951b2844-2765-4954-826b-dec426f39e86 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:09.611326 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-951b2844-2765-4954-826b-dec426f39e86 admin admin] PATH_INFO: `/v3/users/9f85304aa9414cbe9b7435b873003649` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:09.612593 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-951b2844-2765-4954-826b-dec426f39e86 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9f85304aa9414cbe9b7435b873003649)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:09.614613 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-951b2844-2765-4954-826b-dec426f39e86 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:09.665216 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=85,keystone:DELETE=14,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:56:09.681020 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-951b2844-2765-4954-826b-dec426f39e86 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9f85304aa9414cbe9b7435b873003649'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:09.694537 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-951b2844-2765-4954-826b-dec426f39e86 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9f85304aa9414cbe9b7435b873003649'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:09.703477 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-951b2844-2765-4954-826b-dec426f39e86 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9f85304aa9414cbe9b7435b873003649'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:09.712130 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-951b2844-2765-4954-826b-dec426f39e86 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9f85304aa9414cbe9b7435b873003649'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:09.712829 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-951b2844-2765-4954-826b-dec426f39e86 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9f85304aa9414cbe9b7435b873003649'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:09.722323 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1002/2021] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 00:56:09 2022] DELETE /identity/v3/users/9f85304aa9414cbe9b7435b873003649 => generated 0 bytes in 189 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:56:10.089150 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-043846df-fa4d-4556-abf1-02cc6dcde127 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:10.090647 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-043846df-fa4d-4556-abf1-02cc6dcde127 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:56:10.107440 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-043846df-fa4d-4556-abf1-02cc6dcde127 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:10.180400 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ad4ceb3-5f78-4f25-999a-53cbf946689b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:10.181271 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ad4ceb3-5f78-4f25-999a-53cbf946689b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:10.181555 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ad4ceb3-5f78-4f25-999a-53cbf946689b admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:10.181822 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ad4ceb3-5f78-4f25-999a-53cbf946689b admin admin] PATH_INFO: `/v3/projects/7f1d082842b84f289eb2e3b876f41d50` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:10.183066 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ad4ceb3-5f78-4f25-999a-53cbf946689b admin admin] RBAC: Authorizing `identity:delete_project(project_id=7f1d082842b84f289eb2e3b876f41d50)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:10.185328 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ad4ceb3-5f78-4f25-999a-53cbf946689b admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:10.238734 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-8ad4ceb3-5f78-4f25-999a-53cbf946689b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7f1d082842b84f289eb2e3b876f41d50', 'request_id': 'req-8ad4ceb3-5f78-4f25-999a-53cbf946689b'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:10.245455 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-8ad4ceb3-5f78-4f25-999a-53cbf946689b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7f1d082842b84f289eb2e3b876f41d50', 'request_id': 'req-8ad4ceb3-5f78-4f25-999a-53cbf946689b'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:10.252208 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-8ad4ceb3-5f78-4f25-999a-53cbf946689b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7f1d082842b84f289eb2e3b876f41d50', 'request_id': 'req-8ad4ceb3-5f78-4f25-999a-53cbf946689b'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:10.269880 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-8ad4ceb3-5f78-4f25-999a-53cbf946689b admin admin] The token cache is being invalidate because project 7f1d082842b84f289eb2e3b876f41d50 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81508) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 00:56:10.270172 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-8ad4ceb3-5f78-4f25-999a-53cbf946689b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:10.270844 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-8ad4ceb3-5f78-4f25-999a-53cbf946689b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:10.272629 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1020/2022] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 00:56:10 2022] DELETE /identity/v3/projects/7f1d082842b84f289eb2e3b876f41d50 => generated 0 bytes in 184 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:56:10.276828 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-951b2844-2765-4954-826b-dec426f39e86 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:10.278378 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-951b2844-2765-4954-826b-dec426f39e86 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:56:10.298186 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-951b2844-2765-4954-826b-dec426f39e86 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:10.356341 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9a04c51-8047-40d7-8b72-0aa8f695f8e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:10.357002 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9a04c51-8047-40d7-8b72-0aa8f695f8e3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:10.357296 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9a04c51-8047-40d7-8b72-0aa8f695f8e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:10.357573 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9a04c51-8047-40d7-8b72-0aa8f695f8e3 admin admin] PATH_INFO: `/v3/users/be21f3422e4f4abfb1fa71814fedd4bd` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:10.358804 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9a04c51-8047-40d7-8b72-0aa8f695f8e3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=be21f3422e4f4abfb1fa71814fedd4bd)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:10.360749 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9a04c51-8047-40d7-8b72-0aa8f695f8e3 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:10.416408 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-e9a04c51-8047-40d7-8b72-0aa8f695f8e3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'be21f3422e4f4abfb1fa71814fedd4bd'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:10.429613 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-e9a04c51-8047-40d7-8b72-0aa8f695f8e3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'be21f3422e4f4abfb1fa71814fedd4bd'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:10.438590 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-e9a04c51-8047-40d7-8b72-0aa8f695f8e3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'be21f3422e4f4abfb1fa71814fedd4bd'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:10.449395 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-e9a04c51-8047-40d7-8b72-0aa8f695f8e3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'be21f3422e4f4abfb1fa71814fedd4bd'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:10.450419 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-e9a04c51-8047-40d7-8b72-0aa8f695f8e3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'be21f3422e4f4abfb1fa71814fedd4bd'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:10.460273 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1003/2023] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 00:56:10 2022] DELETE /identity/v3/users/be21f3422e4f4abfb1fa71814fedd4bd => generated 0 bytes in 184 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:56:10.763522 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ad4ceb3-5f78-4f25-999a-53cbf946689b admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:10.765097 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-8ad4ceb3-5f78-4f25-999a-53cbf946689b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:56:10.780724 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ad4ceb3-5f78-4f25-999a-53cbf946689b admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:10.842615 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6c26d41-b23b-40e2-bef9-c34efa5f3744 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:10.843257 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c26d41-b23b-40e2-bef9-c34efa5f3744 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:10.843537 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c26d41-b23b-40e2-bef9-c34efa5f3744 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:10.843812 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c26d41-b23b-40e2-bef9-c34efa5f3744 admin admin] PATH_INFO: `/v3/projects/9fdffeebe5f549d296f200d564f2652b` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:10.845010 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6c26d41-b23b-40e2-bef9-c34efa5f3744 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9fdffeebe5f549d296f200d564f2652b)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:10.846993 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6c26d41-b23b-40e2-bef9-c34efa5f3744 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:10.881904 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:56:10.899306 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-d6c26d41-b23b-40e2-bef9-c34efa5f3744 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9fdffeebe5f549d296f200d564f2652b', 'request_id': 'req-d6c26d41-b23b-40e2-bef9-c34efa5f3744'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:10.904766 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-d6c26d41-b23b-40e2-bef9-c34efa5f3744 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9fdffeebe5f549d296f200d564f2652b', 'request_id': 'req-d6c26d41-b23b-40e2-bef9-c34efa5f3744'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:10.912276 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-d6c26d41-b23b-40e2-bef9-c34efa5f3744 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9fdffeebe5f549d296f200d564f2652b', 'request_id': 'req-d6c26d41-b23b-40e2-bef9-c34efa5f3744'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:10.929508 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-d6c26d41-b23b-40e2-bef9-c34efa5f3744 admin admin] The token cache is being invalidate because project 9fdffeebe5f549d296f200d564f2652b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81508) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 00:56:10.929799 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-d6c26d41-b23b-40e2-bef9-c34efa5f3744 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:10.930470 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-d6c26d41-b23b-40e2-bef9-c34efa5f3744 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:10.932001 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1021/2024] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 00:56:10 2022] DELETE /identity/v3/projects/9fdffeebe5f549d296f200d564f2652b => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 28 00:56:10.938066 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01a02980-a410-4577-9f55-e574851049de None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:10.938548 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01a02980-a410-4577-9f55-e574851049de None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:10.938653 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01a02980-a410-4577-9f55-e574851049de None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:10.958730 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-01a02980-a410-4577-9f55-e574851049de None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:56:10.959950 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-01a02980-a410-4577-9f55-e574851049de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:56:10.991496 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=83,keystone:DELETE=15,keystone:INSERT=2 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:56:11.032464 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1004/2025] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:56:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:56:11.040591 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0158efc1-29b1-4787-bbfc-de5cebe88529 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:11.040984 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0158efc1-29b1-4787-bbfc-de5cebe88529 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:11.041482 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0158efc1-29b1-4787-bbfc-de5cebe88529 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:11.067079 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-0158efc1-29b1-4787-bbfc-de5cebe88529 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:56:11.068358 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-0158efc1-29b1-4787-bbfc-de5cebe88529 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:56:11.076798 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1022/2026] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:56:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:56:11.081404 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01a02980-a410-4577-9f55-e574851049de None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:11.091746 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01a02980-a410-4577-9f55-e574851049de None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:11.097130 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ef07137-6a9a-4629-b596-803b6bb335ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:11.097775 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ef07137-6a9a-4629-b596-803b6bb335ac admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:11.098103 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ef07137-6a9a-4629-b596-803b6bb335ac admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:11.098392 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ef07137-6a9a-4629-b596-803b6bb335ac admin admin] PATH_INFO: `/v3/domains` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:11.099081 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ef07137-6a9a-4629-b596-803b6bb335ac admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:56:11.099387 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ef07137-6a9a-4629-b596-803b6bb335ac admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:11.101078 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ef07137-6a9a-4629-b596-803b6bb335ac admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:11.114499 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1005/2027] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Aug 28 00:56:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:56:11.118986 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0158efc1-29b1-4787-bbfc-de5cebe88529 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:11.128871 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0158efc1-29b1-4787-bbfc-de5cebe88529 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:11.135007 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56ec2ba1-6b14-4af1-9441-45b0444611c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:11.135885 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ec2ba1-6b14-4af1-9441-45b0444611c8 admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:11.136341 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ec2ba1-6b14-4af1-9441-45b0444611c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:11.136736 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ec2ba1-6b14-4af1-9441-45b0444611c8 admin admin] PATH_INFO: `/v3/projects` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:11.137590 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56ec2ba1-6b14-4af1-9441-45b0444611c8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:11.139490 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56ec2ba1-6b14-4af1-9441-45b0444611c8 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:11.176079 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1023/2028] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 00:56:11 2022] POST /identity/v3/projects => generated 373 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:56:11.177275 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ef07137-6a9a-4629-b596-803b6bb335ac admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:11.187741 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ef07137-6a9a-4629-b596-803b6bb335ac admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:11.249350 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-773b1073-db4d-4db0-81ea-bb666641430d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:11.249970 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-773b1073-db4d-4db0-81ea-bb666641430d admin admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:11.250309 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-773b1073-db4d-4db0-81ea-bb666641430d admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:11.250506 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-773b1073-db4d-4db0-81ea-bb666641430d admin admin] PATH_INFO: `/v3/users` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:11.251139 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-773b1073-db4d-4db0-81ea-bb666641430d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:11.253135 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-773b1073-db4d-4db0-81ea-bb666641430d admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:11.276907 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1006/2029] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 00:56:11 2022] POST /identity/v3/users => generated 287 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:56:11.281210 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56ec2ba1-6b14-4af1-9441-45b0444611c8 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:11.293461 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56ec2ba1-6b14-4af1-9441-45b0444611c8 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:11.299248 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb9a48fc-b717-48d8-8ddc-12d82d693c77 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:11.299898 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb9a48fc-b717-48d8-8ddc-12d82d693c77 admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:11.300173 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb9a48fc-b717-48d8-8ddc-12d82d693c77 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:11.300440 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb9a48fc-b717-48d8-8ddc-12d82d693c77 admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:11.301143 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb9a48fc-b717-48d8-8ddc-12d82d693c77 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:11.302821 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb9a48fc-b717-48d8-8ddc-12d82d693c77 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:11.314954 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1024/2030] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:56:11 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 00:56:11.319289 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-773b1073-db4d-4db0-81ea-bb666641430d admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:11.331697 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-773b1073-db4d-4db0-81ea-bb666641430d admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:11.337371 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b7f5f16-3053-4ea9-a5ab-edf54cf4d5dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:11.338094 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b7f5f16-3053-4ea9-a5ab-edf54cf4d5dd admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:11.338375 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b7f5f16-3053-4ea9-a5ab-edf54cf4d5dd admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:11.338648 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b7f5f16-3053-4ea9-a5ab-edf54cf4d5dd admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:11.339373 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b7f5f16-3053-4ea9-a5ab-edf54cf4d5dd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:11.340950 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b7f5f16-3053-4ea9-a5ab-edf54cf4d5dd admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:11.352548 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1007/2031] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:56:11 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 00:56:11.357262 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb9a48fc-b717-48d8-8ddc-12d82d693c77 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:11.370605 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb9a48fc-b717-48d8-8ddc-12d82d693c77 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:11.378785 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04f21bd8-638f-4b1f-beae-e4c800f9a0b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:11.379467 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04f21bd8-638f-4b1f-beae-e4c800f9a0b1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:11.379742 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04f21bd8-638f-4b1f-beae-e4c800f9a0b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:11.380016 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04f21bd8-638f-4b1f-beae-e4c800f9a0b1 admin admin] PATH_INFO: `/v3/projects/bae0660dad154f32a8e1df2fd361cc4d/users/efc05d7444f041c8a89a62cb2fd53c39/roles/0487ef7e943e4701915cb8ce0f2899d7` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:11.396875 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04f21bd8-638f-4b1f-beae-e4c800f9a0b1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bae0660dad154f32a8e1df2fd361cc4d, user_id=efc05d7444f041c8a89a62cb2fd53c39, role_id=0487ef7e943e4701915cb8ce0f2899d7)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:11.399586 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04f21bd8-638f-4b1f-beae-e4c800f9a0b1 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:11.413580 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1025/2032] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 00:56:11 2022] PUT /identity/v3/projects/bae0660dad154f32a8e1df2fd361cc4d/users/efc05d7444f041c8a89a62cb2fd53c39/roles/0487ef7e943e4701915cb8ce0f2899d7 => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:56:11.424740 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-320cbe51-3b74-4bd6-bcd9-c6dc2fb786b0 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:11.425153 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-320cbe51-3b74-4bd6-bcd9-c6dc2fb786b0 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:11.425497 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-320cbe51-3b74-4bd6-bcd9-c6dc2fb786b0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:11.456614 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-320cbe51-3b74-4bd6-bcd9-c6dc2fb786b0 None None] MFA Rules not processed for user `efc05d7444f041c8a89a62cb2fd53c39`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:56:11.457891 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-320cbe51-3b74-4bd6-bcd9-c6dc2fb786b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:56:11.531971 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1008/2033] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:56:11 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:56:11.544484 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04f21bd8-638f-4b1f-beae-e4c800f9a0b1 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:11.546648 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-04f21bd8-638f-4b1f-beae-e4c800f9a0b1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:56:11.616921 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04f21bd8-638f-4b1f-beae-e4c800f9a0b1 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:11.621672 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e44faac-6469-418e-aa33-e6d62b1b65da service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:11.622318 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e44faac-6469-418e-aa33-e6d62b1b65da service cinder] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:11.622596 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e44faac-6469-418e-aa33-e6d62b1b65da service cinder] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:11.622872 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e44faac-6469-418e-aa33-e6d62b1b65da service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:11.632681 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e44faac-6469-418e-aa33-e6d62b1b65da service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:11.634198 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e44faac-6469-418e-aa33-e6d62b1b65da service cinder] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:11.643767 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1026/2034] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:56:11 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:56:21.394961 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42ea2175-2b12-4f9a-bb31-fcb963866520 None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:21.395516 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42ea2175-2b12-4f9a-bb31-fcb963866520 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:21.395516 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42ea2175-2b12-4f9a-bb31-fcb963866520 None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:21.396860 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1009/2035] 10.222.0.42 () {54 vars in 921 bytes} [Sun Aug 28 00:56:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:56:21.432130 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3abaa41b-e815-4d2b-afed-bd2891180437 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:21.432661 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3abaa41b-e815-4d2b-afed-bd2891180437 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:21.432661 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3abaa41b-e815-4d2b-afed-bd2891180437 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:21.458367 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-3abaa41b-e815-4d2b-afed-bd2891180437 None None] MFA Rules not processed for user `c2a3834fa6894dada3291e4781bbd553`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:56:21.460054 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-3abaa41b-e815-4d2b-afed-bd2891180437 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:56:21.498501 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42ea2175-2b12-4f9a-bb31-fcb963866520 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:21.501283 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-42ea2175-2b12-4f9a-bb31-fcb963866520 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:56:21.533642 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=92,keystone:DELETE=5,keystone:INSERT=3 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:56:21.539198 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:56:21.552167 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1027/2036] 10.222.0.42 () {58 vars in 1018 bytes} [Sun Aug 28 00:56:21 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:56:21.563482 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3abaa41b-e815-4d2b-afed-bd2891180437 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:21.565263 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-3abaa41b-e815-4d2b-afed-bd2891180437 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:56:21.578644 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42ea2175-2b12-4f9a-bb31-fcb963866520 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:21.578831 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3abaa41b-e815-4d2b-afed-bd2891180437 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:21.583266 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71fc7dc0-77b2-48ed-8a8a-860b73dae8ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:21.583905 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71fc7dc0-77b2-48ed-8a8a-860b73dae8ef admin admin] REQUEST_METHOD: `DELETE` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:21.584038 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d0c3eaa-8b8e-4586-b001-5985722a31ea service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2a3834fa6894dada3291e4781bbd553', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:21.584180 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71fc7dc0-77b2-48ed-8a8a-860b73dae8ef admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:21.584448 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71fc7dc0-77b2-48ed-8a8a-860b73dae8ef admin admin] PATH_INFO: `/v3/users/dd49314743b14266a6a646ffd4d26698` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:21.584647 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d0c3eaa-8b8e-4586-b001-5985722a31ea service placement] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:21.584995 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d0c3eaa-8b8e-4586-b001-5985722a31ea service placement] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:21.585221 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d0c3eaa-8b8e-4586-b001-5985722a31ea service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:21.585606 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71fc7dc0-77b2-48ed-8a8a-860b73dae8ef admin admin] RBAC: Authorizing `identity:delete_user(user_id=dd49314743b14266a6a646ffd4d26698)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:21.587536 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71fc7dc0-77b2-48ed-8a8a-860b73dae8ef admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:21.595380 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d0c3eaa-8b8e-4586-b001-5985722a31ea service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:21.597405 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d0c3eaa-8b8e-4586-b001-5985722a31ea service placement] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:21.625447 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1028/2037] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:56:21 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 62 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 28 00:56:21.655629 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-71fc7dc0-77b2-48ed-8a8a-860b73dae8ef admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'dd49314743b14266a6a646ffd4d26698'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:21.670168 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-71fc7dc0-77b2-48ed-8a8a-860b73dae8ef admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'dd49314743b14266a6a646ffd4d26698'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:21.678217 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-71fc7dc0-77b2-48ed-8a8a-860b73dae8ef admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'dd49314743b14266a6a646ffd4d26698'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:21.687684 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-71fc7dc0-77b2-48ed-8a8a-860b73dae8ef admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'dd49314743b14266a6a646ffd4d26698'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:21.689451 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-71fc7dc0-77b2-48ed-8a8a-860b73dae8ef admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'dd49314743b14266a6a646ffd4d26698'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:21.697693 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1010/2038] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 00:56:21 2022] DELETE /identity/v3/users/dd49314743b14266a6a646ffd4d26698 => generated 0 bytes in 200 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 28 00:56:21.900386 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d0c3eaa-8b8e-4586-b001-5985722a31ea service placement] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:21.915563 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d0c3eaa-8b8e-4586-b001-5985722a31ea service placement] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:21.977026 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-640ff65b-d92c-4b62-b6fb-099b9350250f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:21.977687 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-640ff65b-d92c-4b62-b6fb-099b9350250f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:21.977986 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-640ff65b-d92c-4b62-b6fb-099b9350250f admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:21.978290 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-640ff65b-d92c-4b62-b6fb-099b9350250f admin admin] PATH_INFO: `/v3/projects/d36dd9ee223f496ebc7b6de74b92227a` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:21.979422 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-640ff65b-d92c-4b62-b6fb-099b9350250f admin admin] RBAC: Authorizing `identity:delete_project(project_id=d36dd9ee223f496ebc7b6de74b92227a)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:21.981383 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-640ff65b-d92c-4b62-b6fb-099b9350250f admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:22.034822 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-640ff65b-d92c-4b62-b6fb-099b9350250f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd36dd9ee223f496ebc7b6de74b92227a', 'request_id': 'req-640ff65b-d92c-4b62-b6fb-099b9350250f'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:22.042051 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-640ff65b-d92c-4b62-b6fb-099b9350250f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd36dd9ee223f496ebc7b6de74b92227a', 'request_id': 'req-640ff65b-d92c-4b62-b6fb-099b9350250f'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:22.048544 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-640ff65b-d92c-4b62-b6fb-099b9350250f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd36dd9ee223f496ebc7b6de74b92227a', 'request_id': 'req-640ff65b-d92c-4b62-b6fb-099b9350250f'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:22.062757 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-640ff65b-d92c-4b62-b6fb-099b9350250f admin admin] The token cache is being invalidate because project d36dd9ee223f496ebc7b6de74b92227a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81508) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 00:56:22.062960 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-640ff65b-d92c-4b62-b6fb-099b9350250f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:22.063629 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-640ff65b-d92c-4b62-b6fb-099b9350250f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:22.065819 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1029/2039] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 00:56:21 2022] DELETE /identity/v3/projects/d36dd9ee223f496ebc7b6de74b92227a => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:56:22.070722 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eed7ce34-b76c-4e08-b272-22d9d45e44a9 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:22.071010 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eed7ce34-b76c-4e08-b272-22d9d45e44a9 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:22.071283 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eed7ce34-b76c-4e08-b272-22d9d45e44a9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:22.092252 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-eed7ce34-b76c-4e08-b272-22d9d45e44a9 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:56:22.093495 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-eed7ce34-b76c-4e08-b272-22d9d45e44a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:56:22.162843 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1011/2040] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:56:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:56:22.175047 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcac3e16-6c6c-4af5-88e5-a227ec9d3738 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:22.175591 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcac3e16-6c6c-4af5-88e5-a227ec9d3738 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:22.175984 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcac3e16-6c6c-4af5-88e5-a227ec9d3738 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:22.211346 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-fcac3e16-6c6c-4af5-88e5-a227ec9d3738 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:56:22.212950 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-fcac3e16-6c6c-4af5-88e5-a227ec9d3738 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:56:22.224310 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1030/2041] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:56:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 52 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:56:22.229246 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eed7ce34-b76c-4e08-b272-22d9d45e44a9 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:22.239758 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eed7ce34-b76c-4e08-b272-22d9d45e44a9 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:22.244712 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f09ab29-0705-4abc-86bb-018a3852f947 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:22.245386 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f09ab29-0705-4abc-86bb-018a3852f947 admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:22.245667 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f09ab29-0705-4abc-86bb-018a3852f947 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:22.245934 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f09ab29-0705-4abc-86bb-018a3852f947 admin admin] PATH_INFO: `/v3/domains` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:22.246586 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f09ab29-0705-4abc-86bb-018a3852f947 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:56:22.246888 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f09ab29-0705-4abc-86bb-018a3852f947 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:22.248510 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f09ab29-0705-4abc-86bb-018a3852f947 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:22.261183 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1012/2042] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Aug 28 00:56:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:56:22.266310 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcac3e16-6c6c-4af5-88e5-a227ec9d3738 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:22.278145 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcac3e16-6c6c-4af5-88e5-a227ec9d3738 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:22.283142 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b889e136-fdba-4f8c-b30a-7ede36edb2e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:22.283816 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b889e136-fdba-4f8c-b30a-7ede36edb2e0 admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:22.284089 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b889e136-fdba-4f8c-b30a-7ede36edb2e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:22.284352 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b889e136-fdba-4f8c-b30a-7ede36edb2e0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:22.285050 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b889e136-fdba-4f8c-b30a-7ede36edb2e0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:22.287091 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b889e136-fdba-4f8c-b30a-7ede36edb2e0 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:22.319890 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1031/2043] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 00:56:22 2022] POST /identity/v3/projects => generated 363 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:56:22.321085 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f09ab29-0705-4abc-86bb-018a3852f947 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:22.332483 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f09ab29-0705-4abc-86bb-018a3852f947 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:22.368345 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:56:22.406247 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8e74f71-ef84-4d42-a3de-46fd198c7a88 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:22.406942 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8e74f71-ef84-4d42-a3de-46fd198c7a88 admin admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:22.407235 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8e74f71-ef84-4d42-a3de-46fd198c7a88 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:22.407501 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8e74f71-ef84-4d42-a3de-46fd198c7a88 admin admin] PATH_INFO: `/v3/users` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:22.408164 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8e74f71-ef84-4d42-a3de-46fd198c7a88 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:22.410209 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8e74f71-ef84-4d42-a3de-46fd198c7a88 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:22.435612 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1013/2044] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 00:56:22 2022] POST /identity/v3/users => generated 282 bytes in 115 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:56:22.440305 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b889e136-fdba-4f8c-b30a-7ede36edb2e0 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:22.452504 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b889e136-fdba-4f8c-b30a-7ede36edb2e0 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:22.458365 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc299813-5267-4f72-b57b-135e839b6277 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:22.458981 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc299813-5267-4f72-b57b-135e839b6277 admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:22.459307 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc299813-5267-4f72-b57b-135e839b6277 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:22.459527 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc299813-5267-4f72-b57b-135e839b6277 admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:22.460177 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc299813-5267-4f72-b57b-135e839b6277 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:22.461897 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc299813-5267-4f72-b57b-135e839b6277 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:22.477004 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1032/2045] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:56:22 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 28 00:56:22.478467 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8e74f71-ef84-4d42-a3de-46fd198c7a88 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:22.488945 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8e74f71-ef84-4d42-a3de-46fd198c7a88 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:22.494165 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9064337f-e15e-4510-a60f-28b8cdc941d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:22.494772 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9064337f-e15e-4510-a60f-28b8cdc941d8 admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:22.495088 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9064337f-e15e-4510-a60f-28b8cdc941d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:22.495356 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9064337f-e15e-4510-a60f-28b8cdc941d8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:22.495988 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9064337f-e15e-4510-a60f-28b8cdc941d8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:22.497627 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9064337f-e15e-4510-a60f-28b8cdc941d8 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:22.508249 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1014/2046] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:56:22 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 00:56:22.512330 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc299813-5267-4f72-b57b-135e839b6277 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:22.523402 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc299813-5267-4f72-b57b-135e839b6277 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:22.529462 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab648fef-1b3c-4ad8-b8c7-cb167ef4b8ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:22.530391 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab648fef-1b3c-4ad8-b8c7-cb167ef4b8ed admin admin] REQUEST_METHOD: `PUT` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:22.530848 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab648fef-1b3c-4ad8-b8c7-cb167ef4b8ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:22.531134 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab648fef-1b3c-4ad8-b8c7-cb167ef4b8ed admin admin] PATH_INFO: `/v3/projects/52c58b89487b4b9ab5878c3ed70dc73a/users/906f164607524d8f8e707822d1d09637/roles/0487ef7e943e4701915cb8ce0f2899d7` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:22.549706 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab648fef-1b3c-4ad8-b8c7-cb167ef4b8ed admin admin] RBAC: Authorizing `identity:create_grant(project_id=52c58b89487b4b9ab5878c3ed70dc73a, user_id=906f164607524d8f8e707822d1d09637, role_id=0487ef7e943e4701915cb8ce0f2899d7)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:22.552379 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab648fef-1b3c-4ad8-b8c7-cb167ef4b8ed admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:22.566731 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1033/2047] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 00:56:22 2022] PUT /identity/v3/projects/52c58b89487b4b9ab5878c3ed70dc73a/users/906f164607524d8f8e707822d1d09637/roles/0487ef7e943e4701915cb8ce0f2899d7 => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 28 00:56:22.577480 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7af70c7e-5aee-4ac1-80da-56be41219463 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:22.577790 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7af70c7e-5aee-4ac1-80da-56be41219463 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:22.578068 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7af70c7e-5aee-4ac1-80da-56be41219463 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:22.614081 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-7af70c7e-5aee-4ac1-80da-56be41219463 None None] MFA Rules not processed for user `906f164607524d8f8e707822d1d09637`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:56:22.615777 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-7af70c7e-5aee-4ac1-80da-56be41219463 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:56:22.688427 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1015/2048] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:56:22 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:56:22.693884 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab648fef-1b3c-4ad8-b8c7-cb167ef4b8ed admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:22.700519 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:56:22.779742 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab648fef-1b3c-4ad8-b8c7-cb167ef4b8ed admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:22.785589 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c168cb2-c23f-4c2c-bc1b-77a50edb9b05 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:22.786196 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c168cb2-c23f-4c2c-bc1b-77a50edb9b05 admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:22.786473 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c168cb2-c23f-4c2c-bc1b-77a50edb9b05 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:22.786709 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c168cb2-c23f-4c2c-bc1b-77a50edb9b05 admin admin] PATH_INFO: `/v3/projects` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:22.787364 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c168cb2-c23f-4c2c-bc1b-77a50edb9b05 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:22.789299 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c168cb2-c23f-4c2c-bc1b-77a50edb9b05 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:22.821068 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1034/2049] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 00:56:22 2022] POST /identity/v3/projects => generated 365 bytes in 125 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:56:22.822260 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7af70c7e-5aee-4ac1-80da-56be41219463 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:22.834910 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7af70c7e-5aee-4ac1-80da-56be41219463 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:22.852839 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c168cb2-c23f-4c2c-bc1b-77a50edb9b05 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:22.854701 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-8c168cb2-c23f-4c2c-bc1b-77a50edb9b05 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:56:22.909004 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78307bf5-b27a-40a0-8a09-b8430cf8a5db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:22.909798 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78307bf5-b27a-40a0-8a09-b8430cf8a5db admin admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:22.910200 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78307bf5-b27a-40a0-8a09-b8430cf8a5db admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:22.910752 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78307bf5-b27a-40a0-8a09-b8430cf8a5db admin admin] PATH_INFO: `/v3/users` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:22.911493 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78307bf5-b27a-40a0-8a09-b8430cf8a5db admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:22.913800 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78307bf5-b27a-40a0-8a09-b8430cf8a5db admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:22.929004 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c168cb2-c23f-4c2c-bc1b-77a50edb9b05 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:22.935157 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba686eeb-6e5c-4924-8352-5c3a3805759d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:22.936253 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba686eeb-6e5c-4924-8352-5c3a3805759d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:22.936403 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba686eeb-6e5c-4924-8352-5c3a3805759d admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:22.936786 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba686eeb-6e5c-4924-8352-5c3a3805759d admin admin] PATH_INFO: `/v3/users/efc05d7444f041c8a89a62cb2fd53c39` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:22.938119 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba686eeb-6e5c-4924-8352-5c3a3805759d admin admin] RBAC: Authorizing `identity:delete_user(user_id=efc05d7444f041c8a89a62cb2fd53c39)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:22.940257 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba686eeb-6e5c-4924-8352-5c3a3805759d admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:22.945423 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1016/2050] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 00:56:22 2022] POST /identity/v3/users => generated 289 bytes in 121 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:56:22.946726 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78307bf5-b27a-40a0-8a09-b8430cf8a5db admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:22.970640 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78307bf5-b27a-40a0-8a09-b8430cf8a5db admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:22.975673 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30d5e622-cda9-4001-8a8a-7b0852b2dae9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:22.976279 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30d5e622-cda9-4001-8a8a-7b0852b2dae9 admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:22.976567 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30d5e622-cda9-4001-8a8a-7b0852b2dae9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:22.976841 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30d5e622-cda9-4001-8a8a-7b0852b2dae9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:22.977515 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30d5e622-cda9-4001-8a8a-7b0852b2dae9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:22.979136 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30d5e622-cda9-4001-8a8a-7b0852b2dae9 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:22.992824 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1017/2051] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:56:22 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 28 00:56:22.994982 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30d5e622-cda9-4001-8a8a-7b0852b2dae9 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:22.997587 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-ba686eeb-6e5c-4924-8352-5c3a3805759d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'efc05d7444f041c8a89a62cb2fd53c39'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:22.998244 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-ba686eeb-6e5c-4924-8352-5c3a3805759d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'efc05d7444f041c8a89a62cb2fd53c39'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:23.006443 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30d5e622-cda9-4001-8a8a-7b0852b2dae9 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:23.008528 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-ba686eeb-6e5c-4924-8352-5c3a3805759d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'efc05d7444f041c8a89a62cb2fd53c39'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:23.011536 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45ecf38c-86f1-4aff-b9be-2c02dd034d18 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:23.012194 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45ecf38c-86f1-4aff-b9be-2c02dd034d18 admin admin] REQUEST_METHOD: `PUT` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:23.012473 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45ecf38c-86f1-4aff-b9be-2c02dd034d18 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:23.012745 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45ecf38c-86f1-4aff-b9be-2c02dd034d18 admin admin] PATH_INFO: `/v3/projects/7e6426b4b5f6411a97f2fd6c25045efc/users/27adb14735fa47a789b8f98c24048839/roles/f08d6043261747e3b4763c744eef7dfb` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:23.016449 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-ba686eeb-6e5c-4924-8352-5c3a3805759d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'efc05d7444f041c8a89a62cb2fd53c39'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:23.016609 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-ba686eeb-6e5c-4924-8352-5c3a3805759d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'efc05d7444f041c8a89a62cb2fd53c39'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:23.027686 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45ecf38c-86f1-4aff-b9be-2c02dd034d18 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7e6426b4b5f6411a97f2fd6c25045efc, user_id=27adb14735fa47a789b8f98c24048839, role_id=f08d6043261747e3b4763c744eef7dfb)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:23.030653 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45ecf38c-86f1-4aff-b9be-2c02dd034d18 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:23.034043 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:56:23.035580 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1035/2052] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 00:56:22 2022] DELETE /identity/v3/users/efc05d7444f041c8a89a62cb2fd53c39 => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 28 00:56:23.041118 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23fc57fb-b4ef-4f15-8794-827275c7a500 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:23.041392 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23fc57fb-b4ef-4f15-8794-827275c7a500 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:23.041664 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23fc57fb-b4ef-4f15-8794-827275c7a500 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:23.047058 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1018/2053] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 00:56:22 2022] PUT /identity/v3/projects/7e6426b4b5f6411a97f2fd6c25045efc/users/27adb14735fa47a789b8f98c24048839/roles/f08d6043261747e3b4763c744eef7dfb => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 28 00:56:23.050956 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45ecf38c-86f1-4aff-b9be-2c02dd034d18 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:23.052606 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-45ecf38c-86f1-4aff-b9be-2c02dd034d18 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:56:23.062051 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-23fc57fb-b4ef-4f15-8794-827275c7a500 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:56:23.063254 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-23fc57fb-b4ef-4f15-8794-827275c7a500 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:56:23.067533 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:56:23.121185 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45ecf38c-86f1-4aff-b9be-2c02dd034d18 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:23.124278 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c4be0e1-5c62-45eb-ae92-8b0a8f545ef3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:23.124977 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c4be0e1-5c62-45eb-ae92-8b0a8f545ef3 admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:23.125242 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c4be0e1-5c62-45eb-ae92-8b0a8f545ef3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:23.125537 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c4be0e1-5c62-45eb-ae92-8b0a8f545ef3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:23.126277 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c4be0e1-5c62-45eb-ae92-8b0a8f545ef3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:23.127920 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c4be0e1-5c62-45eb-ae92-8b0a8f545ef3 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:23.129629 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1036/2054] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:56:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:56:23.141837 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1019/2055] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:56:23 2022] GET /identity/v3/roles => generated 1420 bytes in 88 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 28 00:56:23.142003 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23fc57fb-b4ef-4f15-8794-827275c7a500 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:23.142003 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-23fc57fb-b4ef-4f15-8794-827275c7a500 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:56:23.142936 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c4be0e1-5c62-45eb-ae92-8b0a8f545ef3 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:23.153181 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c4be0e1-5c62-45eb-ae92-8b0a8f545ef3 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:23.158426 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05a69b85-cd91-4831-8daa-65e12b66cfd2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:23.159075 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05a69b85-cd91-4831-8daa-65e12b66cfd2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:23.159350 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05a69b85-cd91-4831-8daa-65e12b66cfd2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:23.159613 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05a69b85-cd91-4831-8daa-65e12b66cfd2 admin admin] PATH_INFO: `/v3/projects/7e6426b4b5f6411a97f2fd6c25045efc/users/27adb14735fa47a789b8f98c24048839/roles/f08d6043261747e3b4763c744eef7dfb` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:23.161676 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05a69b85-cd91-4831-8daa-65e12b66cfd2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7e6426b4b5f6411a97f2fd6c25045efc, user_id=27adb14735fa47a789b8f98c24048839, role_id=f08d6043261747e3b4763c744eef7dfb)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:23.164403 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05a69b85-cd91-4831-8daa-65e12b66cfd2 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:23.175052 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1020/2056] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 00:56:23 2022] PUT /identity/v3/projects/7e6426b4b5f6411a97f2fd6c25045efc/users/27adb14735fa47a789b8f98c24048839/roles/f08d6043261747e3b4763c744eef7dfb => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:56:23.187338 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85c8abb3-c287-4318-a417-fa194a61b1bc None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:23.187623 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85c8abb3-c287-4318-a417-fa194a61b1bc None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:23.187911 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85c8abb3-c287-4318-a417-fa194a61b1bc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:23.207572 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23fc57fb-b4ef-4f15-8794-827275c7a500 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:23.219383 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-85c8abb3-c287-4318-a417-fa194a61b1bc None None] MFA Rules not processed for user `27adb14735fa47a789b8f98c24048839`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:56:23.220673 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-85c8abb3-c287-4318-a417-fa194a61b1bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:56:23.259884 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43f93a2f-46ab-4c9b-ac72-8fb188cfa35a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:23.260527 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43f93a2f-46ab-4c9b-ac72-8fb188cfa35a service neutron] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:23.260793 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43f93a2f-46ab-4c9b-ac72-8fb188cfa35a service neutron] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:23.261095 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43f93a2f-46ab-4c9b-ac72-8fb188cfa35a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:23.297966 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1021/2057] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:56:23 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:56:23.309820 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85c8abb3-c287-4318-a417-fa194a61b1bc None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:23.311230 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-85c8abb3-c287-4318-a417-fa194a61b1bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:56:23.327708 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43f93a2f-46ab-4c9b-ac72-8fb188cfa35a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:23.329501 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43f93a2f-46ab-4c9b-ac72-8fb188cfa35a service neutron] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:23.339589 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1037/2058] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:56:23 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 201 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 28 00:56:23.373576 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85c8abb3-c287-4318-a417-fa194a61b1bc None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:23.377664 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5df01dc-3bb7-48de-84db-30ac00ad37a9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:23.378279 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5df01dc-3bb7-48de-84db-30ac00ad37a9 service cinder] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:23.378559 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5df01dc-3bb7-48de-84db-30ac00ad37a9 service cinder] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:23.378920 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5df01dc-3bb7-48de-84db-30ac00ad37a9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:23.391106 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5df01dc-3bb7-48de-84db-30ac00ad37a9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:23.392630 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5df01dc-3bb7-48de-84db-30ac00ad37a9 service cinder] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:23.403490 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1022/2059] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:56:23 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 28 00:56:23.631936 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43f93a2f-46ab-4c9b-ac72-8fb188cfa35a service neutron] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:23.633699 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-43f93a2f-46ab-4c9b-ac72-8fb188cfa35a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:56:23.651407 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43f93a2f-46ab-4c9b-ac72-8fb188cfa35a service neutron] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:23.658681 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8583fe43-bb56-4fc7-9ec6-f1425881dd02 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:23.660368 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8583fe43-bb56-4fc7-9ec6-f1425881dd02 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:23.660689 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8583fe43-bb56-4fc7-9ec6-f1425881dd02 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:23.660987 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8583fe43-bb56-4fc7-9ec6-f1425881dd02 admin admin] PATH_INFO: `/v3/projects/bae0660dad154f32a8e1df2fd361cc4d` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:23.662731 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8583fe43-bb56-4fc7-9ec6-f1425881dd02 admin admin] RBAC: Authorizing `identity:delete_project(project_id=bae0660dad154f32a8e1df2fd361cc4d)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:23.664696 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8583fe43-bb56-4fc7-9ec6-f1425881dd02 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:23.681157 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:56:23.717617 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-8583fe43-bb56-4fc7-9ec6-f1425881dd02 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bae0660dad154f32a8e1df2fd361cc4d', 'request_id': 'req-8583fe43-bb56-4fc7-9ec6-f1425881dd02'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:23.724333 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-8583fe43-bb56-4fc7-9ec6-f1425881dd02 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bae0660dad154f32a8e1df2fd361cc4d', 'request_id': 'req-8583fe43-bb56-4fc7-9ec6-f1425881dd02'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:23.733590 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-8583fe43-bb56-4fc7-9ec6-f1425881dd02 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bae0660dad154f32a8e1df2fd361cc4d', 'request_id': 'req-8583fe43-bb56-4fc7-9ec6-f1425881dd02'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:23.749490 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-8583fe43-bb56-4fc7-9ec6-f1425881dd02 admin admin] The token cache is being invalidate because project bae0660dad154f32a8e1df2fd361cc4d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81508) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 00:56:23.749964 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-8583fe43-bb56-4fc7-9ec6-f1425881dd02 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:23.750681 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-8583fe43-bb56-4fc7-9ec6-f1425881dd02 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:23.752688 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1038/2060] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 00:56:23 2022] DELETE /identity/v3/projects/bae0660dad154f32a8e1df2fd361cc4d => generated 0 bytes in 121 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:56:23.758977 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2631d1c-1975-43bb-99ad-ab372016253a None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:23.759521 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2631d1c-1975-43bb-99ad-ab372016253a None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:23.759521 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2631d1c-1975-43bb-99ad-ab372016253a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:23.776305 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:56:23.784995 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-a2631d1c-1975-43bb-99ad-ab372016253a None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:56:23.784995 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-a2631d1c-1975-43bb-99ad-ab372016253a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:56:23.855066 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1023/2061] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:56:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:56:23.890785 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8583fe43-bb56-4fc7-9ec6-f1425881dd02 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:23.892758 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-8583fe43-bb56-4fc7-9ec6-f1425881dd02 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:56:23.907124 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8583fe43-bb56-4fc7-9ec6-f1425881dd02 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:23.912631 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e89f01b5-02f7-47e0-9618-883b0eb12f6a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:23.913348 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e89f01b5-02f7-47e0-9618-883b0eb12f6a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:23.913625 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e89f01b5-02f7-47e0-9618-883b0eb12f6a admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:23.913896 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e89f01b5-02f7-47e0-9618-883b0eb12f6a admin admin] PATH_INFO: `/v3/users/906f164607524d8f8e707822d1d09637` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:23.915170 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e89f01b5-02f7-47e0-9618-883b0eb12f6a admin admin] RBAC: Authorizing `identity:delete_user(user_id=906f164607524d8f8e707822d1d09637)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:23.917203 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e89f01b5-02f7-47e0-9618-883b0eb12f6a admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:23.925262 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3693eef4-c0db-4960-b2e2-19072e2bc215 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:23.925553 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3693eef4-c0db-4960-b2e2-19072e2bc215 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:23.925825 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3693eef4-c0db-4960-b2e2-19072e2bc215 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:23.946978 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-3693eef4-c0db-4960-b2e2-19072e2bc215 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:56:23.948477 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-3693eef4-c0db-4960-b2e2-19072e2bc215 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:56:23.960278 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1024/2062] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:56:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 28 00:56:23.965007 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3693eef4-c0db-4960-b2e2-19072e2bc215 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:23.972740 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-e89f01b5-02f7-47e0-9618-883b0eb12f6a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '906f164607524d8f8e707822d1d09637'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:23.973588 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-e89f01b5-02f7-47e0-9618-883b0eb12f6a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '906f164607524d8f8e707822d1d09637'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:23.975414 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3693eef4-c0db-4960-b2e2-19072e2bc215 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:23.981160 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87a99e98-af75-4058-bba7-a485c7619b6b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:23.981993 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a99e98-af75-4058-bba7-a485c7619b6b admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:23.982270 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a99e98-af75-4058-bba7-a485c7619b6b admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:23.982811 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a99e98-af75-4058-bba7-a485c7619b6b admin admin] PATH_INFO: `/v3/domains` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:23.982918 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-e89f01b5-02f7-47e0-9618-883b0eb12f6a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '906f164607524d8f8e707822d1d09637'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:23.983587 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87a99e98-af75-4058-bba7-a485c7619b6b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:56:23.984007 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87a99e98-af75-4058-bba7-a485c7619b6b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:23.985592 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87a99e98-af75-4058-bba7-a485c7619b6b admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:23.993959 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-e89f01b5-02f7-47e0-9618-883b0eb12f6a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '906f164607524d8f8e707822d1d09637'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:23.994703 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-e89f01b5-02f7-47e0-9618-883b0eb12f6a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '906f164607524d8f8e707822d1d09637'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:23.998410 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1025/2063] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Aug 28 00:56:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:56:24.003232 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87a99e98-af75-4058-bba7-a485c7619b6b admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:24.005559 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-87a99e98-af75-4058-bba7-a485c7619b6b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:56:24.014874 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1039/2064] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 00:56:23 2022] DELETE /identity/v3/users/906f164607524d8f8e707822d1d09637 => generated 0 bytes in 124 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:56:24.019215 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa1539e2-177f-499c-ac41-885725b310aa None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:24.019504 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa1539e2-177f-499c-ac41-885725b310aa None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:24.019760 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87a99e98-af75-4058-bba7-a485c7619b6b admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:24.019875 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa1539e2-177f-499c-ac41-885725b310aa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:24.048837 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-fa1539e2-177f-499c-ac41-885725b310aa None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:56:24.050611 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-fa1539e2-177f-499c-ac41-885725b310aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:56:24.084217 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64c3e62e-6618-417d-b155-8df304b5f647 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:24.084932 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64c3e62e-6618-417d-b155-8df304b5f647 admin admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:24.085302 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64c3e62e-6618-417d-b155-8df304b5f647 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:24.085644 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64c3e62e-6618-417d-b155-8df304b5f647 admin admin] PATH_INFO: `/v3/projects` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:24.086376 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64c3e62e-6618-417d-b155-8df304b5f647 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:24.088426 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64c3e62e-6618-417d-b155-8df304b5f647 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:24.124278 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1026/2065] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 00:56:24 2022] POST /identity/v3/projects => generated 391 bytes in 122 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:56:24.128448 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64c3e62e-6618-417d-b155-8df304b5f647 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:24.137955 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1040/2066] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:56:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 28 00:56:24.145571 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa1539e2-177f-499c-ac41-885725b310aa None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:24.155850 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-fa1539e2-177f-499c-ac41-885725b310aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:56:24.161391 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:56:24.203236 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64c3e62e-6618-417d-b155-8df304b5f647 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:24.207419 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa1539e2-177f-499c-ac41-885725b310aa None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:24.208176 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b8fa884-cb9c-48b9-842c-971169296416 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:24.208747 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b8fa884-cb9c-48b9-842c-971169296416 admin admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:24.209058 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b8fa884-cb9c-48b9-842c-971169296416 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:24.209331 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b8fa884-cb9c-48b9-842c-971169296416 admin admin] PATH_INFO: `/v3/users` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:24.209967 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b8fa884-cb9c-48b9-842c-971169296416 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:24.211893 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b8fa884-cb9c-48b9-842c-971169296416 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:24.235305 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1027/2067] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 00:56:24 2022] POST /identity/v3/users => generated 296 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 28 00:56:24.239595 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b8fa884-cb9c-48b9-842c-971169296416 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:24.244830 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:56:24.250833 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b8fa884-cb9c-48b9-842c-971169296416 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:24.255966 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cfb9853-e47e-462c-b86c-c32adb9d68ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:24.256571 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cfb9853-e47e-462c-b86c-c32adb9d68ea admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:24.256852 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cfb9853-e47e-462c-b86c-c32adb9d68ea admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:24.257156 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cfb9853-e47e-462c-b86c-c32adb9d68ea admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:24.257811 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cfb9853-e47e-462c-b86c-c32adb9d68ea admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:24.259456 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cfb9853-e47e-462c-b86c-c32adb9d68ea admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:24.261552 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-469dc7a6-b186-4380-baac-313cd55b8c54 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:24.262164 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-469dc7a6-b186-4380-baac-313cd55b8c54 service neutron] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:24.262479 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-469dc7a6-b186-4380-baac-313cd55b8c54 service neutron] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:24.263059 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-469dc7a6-b186-4380-baac-313cd55b8c54 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:24.270192 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1028/2068] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:56:24 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 28 00:56:24.272595 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-469dc7a6-b186-4380-baac-313cd55b8c54 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:24.273989 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cfb9853-e47e-462c-b86c-c32adb9d68ea admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:24.274169 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-469dc7a6-b186-4380-baac-313cd55b8c54 service neutron] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:24.283732 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cfb9853-e47e-462c-b86c-c32adb9d68ea admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:24.284018 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1041/2069] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:56:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 139 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 28 00:56:24.288246 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd4e4b12-92f9-434b-9299-8b8e9b3a0edf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:24.288847 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd4e4b12-92f9-434b-9299-8b8e9b3a0edf admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:24.289155 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd4e4b12-92f9-434b-9299-8b8e9b3a0edf admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:24.289423 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd4e4b12-92f9-434b-9299-8b8e9b3a0edf admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:24.290056 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd4e4b12-92f9-434b-9299-8b8e9b3a0edf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:24.291641 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd4e4b12-92f9-434b-9299-8b8e9b3a0edf admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:24.302420 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1029/2070] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:56:24 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 28 00:56:24.306754 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-469dc7a6-b186-4380-baac-313cd55b8c54 service neutron] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:24.317784 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-469dc7a6-b186-4380-baac-313cd55b8c54 service neutron] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:24.324876 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe5519f1-96aa-4f31-ab7d-de820452a57f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:24.325901 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe5519f1-96aa-4f31-ab7d-de820452a57f admin admin] REQUEST_METHOD: `PUT` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:24.326192 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe5519f1-96aa-4f31-ab7d-de820452a57f admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:24.326463 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe5519f1-96aa-4f31-ab7d-de820452a57f admin admin] PATH_INFO: `/v3/projects/da2beaaf1c7846a5aac4a75fd04461ce/users/dec85926b25142529c9ba1fb686dba03/roles/0487ef7e943e4701915cb8ce0f2899d7` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:24.344224 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe5519f1-96aa-4f31-ab7d-de820452a57f admin admin] RBAC: Authorizing `identity:create_grant(project_id=da2beaaf1c7846a5aac4a75fd04461ce, user_id=dec85926b25142529c9ba1fb686dba03, role_id=0487ef7e943e4701915cb8ce0f2899d7)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:24.348008 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe5519f1-96aa-4f31-ab7d-de820452a57f admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:24.363340 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1042/2071] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 00:56:24 2022] PUT /identity/v3/projects/da2beaaf1c7846a5aac4a75fd04461ce/users/dec85926b25142529c9ba1fb686dba03/roles/0487ef7e943e4701915cb8ce0f2899d7 => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:56:24.373896 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ec69b9a-10fa-4ddf-a562-c9fe9c4f362a None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:24.374202 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ec69b9a-10fa-4ddf-a562-c9fe9c4f362a None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:24.374472 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ec69b9a-10fa-4ddf-a562-c9fe9c4f362a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:24.402628 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-9ec69b9a-10fa-4ddf-a562-c9fe9c4f362a None None] MFA Rules not processed for user `dec85926b25142529c9ba1fb686dba03`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:56:24.403869 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-9ec69b9a-10fa-4ddf-a562-c9fe9c4f362a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:56:24.473630 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1030/2072] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:56:24 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:56:24.484551 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe5519f1-96aa-4f31-ab7d-de820452a57f admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:24.486514 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-fe5519f1-96aa-4f31-ab7d-de820452a57f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:56:24.547409 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe5519f1-96aa-4f31-ab7d-de820452a57f admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:24.551444 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f88dd6ed-3152-4bd3-9e08-9b5211315f13 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:24.552107 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f88dd6ed-3152-4bd3-9e08-9b5211315f13 service cinder] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:24.552356 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f88dd6ed-3152-4bd3-9e08-9b5211315f13 service cinder] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:24.552626 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f88dd6ed-3152-4bd3-9e08-9b5211315f13 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:24.562247 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f88dd6ed-3152-4bd3-9e08-9b5211315f13 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:24.564106 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f88dd6ed-3152-4bd3-9e08-9b5211315f13 service cinder] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:24.571219 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ec69b9a-10fa-4ddf-a562-c9fe9c4f362a None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:24.572682 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-9ec69b9a-10fa-4ddf-a562-c9fe9c4f362a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:56:24.575580 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1043/2073] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:56:24 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:56:24.656570 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ec69b9a-10fa-4ddf-a562-c9fe9c4f362a None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:24.661644 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fa240b4-4338-49b4-85f3-2fce449af3d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:24.662292 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fa240b4-4338-49b4-85f3-2fce449af3d8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:24.662565 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fa240b4-4338-49b4-85f3-2fce449af3d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:24.662837 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fa240b4-4338-49b4-85f3-2fce449af3d8 admin admin] PATH_INFO: `/v3/projects/52c58b89487b4b9ab5878c3ed70dc73a` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:24.664034 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fa240b4-4338-49b4-85f3-2fce449af3d8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=52c58b89487b4b9ab5878c3ed70dc73a)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:24.666140 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fa240b4-4338-49b4-85f3-2fce449af3d8 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:24.718131 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-1fa240b4-4338-49b4-85f3-2fce449af3d8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '52c58b89487b4b9ab5878c3ed70dc73a', 'request_id': 'req-1fa240b4-4338-49b4-85f3-2fce449af3d8'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:24.724806 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-1fa240b4-4338-49b4-85f3-2fce449af3d8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '52c58b89487b4b9ab5878c3ed70dc73a', 'request_id': 'req-1fa240b4-4338-49b4-85f3-2fce449af3d8'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:24.731770 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-1fa240b4-4338-49b4-85f3-2fce449af3d8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '52c58b89487b4b9ab5878c3ed70dc73a', 'request_id': 'req-1fa240b4-4338-49b4-85f3-2fce449af3d8'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:24.744993 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:56:24.749622 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-1fa240b4-4338-49b4-85f3-2fce449af3d8 admin admin] The token cache is being invalidate because project 52c58b89487b4b9ab5878c3ed70dc73a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81507) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 00:56:24.749921 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-1fa240b4-4338-49b4-85f3-2fce449af3d8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:24.750795 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-1fa240b4-4338-49b4-85f3-2fce449af3d8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:24.753232 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1031/2074] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 00:56:24 2022] DELETE /identity/v3/projects/52c58b89487b4b9ab5878c3ed70dc73a => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:56:24.757478 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f88dd6ed-3152-4bd3-9e08-9b5211315f13 service cinder] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:24.771219 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f88dd6ed-3152-4bd3-9e08-9b5211315f13 service cinder] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:24.820614 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:56:24.836030 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17e7ff90-4cd7-4968-a835-5c7bba355f50 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:24.836668 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17e7ff90-4cd7-4968-a835-5c7bba355f50 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:24.836947 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17e7ff90-4cd7-4968-a835-5c7bba355f50 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:24.837254 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17e7ff90-4cd7-4968-a835-5c7bba355f50 admin admin] PATH_INFO: `/v3/users/27adb14735fa47a789b8f98c24048839` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:24.838386 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17e7ff90-4cd7-4968-a835-5c7bba355f50 admin admin] RBAC: Authorizing `identity:delete_user(user_id=27adb14735fa47a789b8f98c24048839)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:24.840304 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17e7ff90-4cd7-4968-a835-5c7bba355f50 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:24.910605 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-17e7ff90-4cd7-4968-a835-5c7bba355f50 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '27adb14735fa47a789b8f98c24048839'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:24.911388 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-17e7ff90-4cd7-4968-a835-5c7bba355f50 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '27adb14735fa47a789b8f98c24048839'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:24.921811 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-17e7ff90-4cd7-4968-a835-5c7bba355f50 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '27adb14735fa47a789b8f98c24048839'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:24.929272 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-17e7ff90-4cd7-4968-a835-5c7bba355f50 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '27adb14735fa47a789b8f98c24048839'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:24.930000 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-17e7ff90-4cd7-4968-a835-5c7bba355f50 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '27adb14735fa47a789b8f98c24048839'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:24.947363 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1044/2075] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 00:56:24 2022] DELETE /identity/v3/users/27adb14735fa47a789b8f98c24048839 => generated 0 bytes in 190 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:56:25.246755 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fa240b4-4338-49b4-85f3-2fce449af3d8 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:25.248226 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-1fa240b4-4338-49b4-85f3-2fce449af3d8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:56:25.263823 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fa240b4-4338-49b4-85f3-2fce449af3d8 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:25.327421 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8416d93a-a89a-41c6-b076-05ce85b84e20 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:25.328601 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8416d93a-a89a-41c6-b076-05ce85b84e20 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:25.328601 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8416d93a-a89a-41c6-b076-05ce85b84e20 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:25.328909 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8416d93a-a89a-41c6-b076-05ce85b84e20 admin admin] PATH_INFO: `/v3/projects/7e6426b4b5f6411a97f2fd6c25045efc` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:25.330354 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8416d93a-a89a-41c6-b076-05ce85b84e20 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7e6426b4b5f6411a97f2fd6c25045efc)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:25.332509 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8416d93a-a89a-41c6-b076-05ce85b84e20 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:25.392867 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-8416d93a-a89a-41c6-b076-05ce85b84e20 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7e6426b4b5f6411a97f2fd6c25045efc', 'request_id': 'req-8416d93a-a89a-41c6-b076-05ce85b84e20'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:25.398452 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-8416d93a-a89a-41c6-b076-05ce85b84e20 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7e6426b4b5f6411a97f2fd6c25045efc', 'request_id': 'req-8416d93a-a89a-41c6-b076-05ce85b84e20'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:25.405037 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-8416d93a-a89a-41c6-b076-05ce85b84e20 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7e6426b4b5f6411a97f2fd6c25045efc', 'request_id': 'req-8416d93a-a89a-41c6-b076-05ce85b84e20'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:25.419535 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-8416d93a-a89a-41c6-b076-05ce85b84e20 admin admin] The token cache is being invalidate because project 7e6426b4b5f6411a97f2fd6c25045efc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81507) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 00:56:25.419718 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-8416d93a-a89a-41c6-b076-05ce85b84e20 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:25.420397 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-8416d93a-a89a-41c6-b076-05ce85b84e20 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:25.421971 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1032/2076] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 00:56:25 2022] DELETE /identity/v3/projects/7e6426b4b5f6411a97f2fd6c25045efc => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:56:25.428326 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec930af0-66ba-465a-b8c7-fa88fde1e4b5 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:25.428615 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec930af0-66ba-465a-b8c7-fa88fde1e4b5 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:25.428892 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec930af0-66ba-465a-b8c7-fa88fde1e4b5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:25.449810 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-ec930af0-66ba-465a-b8c7-fa88fde1e4b5 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:56:25.451018 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-ec930af0-66ba-465a-b8c7-fa88fde1e4b5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:56:25.520752 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1045/2077] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:56:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:56:25.529296 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4efc0632-8b26-4442-b5d8-96517e779880 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:25.529716 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4efc0632-8b26-4442-b5d8-96517e779880 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:25.530156 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4efc0632-8b26-4442-b5d8-96517e779880 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:25.552635 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-4efc0632-8b26-4442-b5d8-96517e779880 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:56:25.553880 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-4efc0632-8b26-4442-b5d8-96517e779880 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:56:25.563144 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1033/2078] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:56:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:56:25.567985 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec930af0-66ba-465a-b8c7-fa88fde1e4b5 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:25.579095 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec930af0-66ba-465a-b8c7-fa88fde1e4b5 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:25.584641 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30da2b0e-2a8b-4787-bea9-f040f52dba9c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:25.585277 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30da2b0e-2a8b-4787-bea9-f040f52dba9c admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:25.585572 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30da2b0e-2a8b-4787-bea9-f040f52dba9c admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:25.585846 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30da2b0e-2a8b-4787-bea9-f040f52dba9c admin admin] PATH_INFO: `/v3/domains` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:25.586498 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30da2b0e-2a8b-4787-bea9-f040f52dba9c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:56:25.586799 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30da2b0e-2a8b-4787-bea9-f040f52dba9c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:25.588417 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30da2b0e-2a8b-4787-bea9-f040f52dba9c admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:25.601959 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1046/2079] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Aug 28 00:56:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:56:25.606422 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4efc0632-8b26-4442-b5d8-96517e779880 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:25.616008 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4efc0632-8b26-4442-b5d8-96517e779880 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:25.620731 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4563c1ef-01fc-471f-828d-dc80781191d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:25.621358 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4563c1ef-01fc-471f-828d-dc80781191d7 admin admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:25.621638 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4563c1ef-01fc-471f-828d-dc80781191d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:25.621910 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4563c1ef-01fc-471f-828d-dc80781191d7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:25.622536 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4563c1ef-01fc-471f-828d-dc80781191d7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:25.624446 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4563c1ef-01fc-471f-828d-dc80781191d7 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:25.654997 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1034/2080] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 00:56:25 2022] POST /identity/v3/projects => generated 369 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:56:25.659956 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30da2b0e-2a8b-4787-bea9-f040f52dba9c admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:25.670207 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30da2b0e-2a8b-4787-bea9-f040f52dba9c admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:25.722061 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:56:25.731133 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa610e30-229c-4847-80d4-5274950780e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:25.731816 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa610e30-229c-4847-80d4-5274950780e4 admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:25.732482 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa610e30-229c-4847-80d4-5274950780e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:25.732991 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa610e30-229c-4847-80d4-5274950780e4 admin admin] PATH_INFO: `/v3/users` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:25.734129 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa610e30-229c-4847-80d4-5274950780e4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:25.736503 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa610e30-229c-4847-80d4-5274950780e4 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:25.760188 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1047/2081] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 00:56:25 2022] POST /identity/v3/users => generated 285 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:56:25.765268 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4563c1ef-01fc-471f-828d-dc80781191d7 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:25.776572 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4563c1ef-01fc-471f-828d-dc80781191d7 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:25.782391 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ebabfad-8137-4d25-a36c-2a47b6614bc7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:25.782988 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ebabfad-8137-4d25-a36c-2a47b6614bc7 admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:25.783266 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ebabfad-8137-4d25-a36c-2a47b6614bc7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:25.783534 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ebabfad-8137-4d25-a36c-2a47b6614bc7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:25.784204 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ebabfad-8137-4d25-a36c-2a47b6614bc7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:25.785887 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ebabfad-8137-4d25-a36c-2a47b6614bc7 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:25.796304 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1035/2082] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:56:25 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 00:56:25.800708 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa610e30-229c-4847-80d4-5274950780e4 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:25.811915 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa610e30-229c-4847-80d4-5274950780e4 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:25.816985 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdc847f7-cf69-4ee5-be82-fd80d992a5ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:25.817577 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdc847f7-cf69-4ee5-be82-fd80d992a5ac admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:25.817854 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdc847f7-cf69-4ee5-be82-fd80d992a5ac admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:25.818124 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdc847f7-cf69-4ee5-be82-fd80d992a5ac admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:25.818766 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdc847f7-cf69-4ee5-be82-fd80d992a5ac admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:25.820401 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdc847f7-cf69-4ee5-be82-fd80d992a5ac admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:25.832976 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1048/2083] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:56:25 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 00:56:25.837322 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ebabfad-8137-4d25-a36c-2a47b6614bc7 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:25.848153 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ebabfad-8137-4d25-a36c-2a47b6614bc7 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:25.853877 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-180604a6-5bba-4943-87e6-227ae0a09725 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:25.854529 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-180604a6-5bba-4943-87e6-227ae0a09725 admin admin] REQUEST_METHOD: `PUT` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:25.854808 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-180604a6-5bba-4943-87e6-227ae0a09725 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:25.855076 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-180604a6-5bba-4943-87e6-227ae0a09725 admin admin] PATH_INFO: `/v3/projects/d39934e430a649a7873c1010ecff4901/users/53ea7d8704204c778d402e70dacb9bf9/roles/0487ef7e943e4701915cb8ce0f2899d7` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:25.872795 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-180604a6-5bba-4943-87e6-227ae0a09725 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d39934e430a649a7873c1010ecff4901, user_id=53ea7d8704204c778d402e70dacb9bf9, role_id=0487ef7e943e4701915cb8ce0f2899d7)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:25.875479 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-180604a6-5bba-4943-87e6-227ae0a09725 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:25.889754 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1036/2084] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 00:56:25 2022] PUT /identity/v3/projects/d39934e430a649a7873c1010ecff4901/users/53ea7d8704204c778d402e70dacb9bf9/roles/0487ef7e943e4701915cb8ce0f2899d7 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:56:25.899920 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea462059-97dd-4cff-83ec-bbfbe75e0630 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:25.900203 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea462059-97dd-4cff-83ec-bbfbe75e0630 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:25.900470 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea462059-97dd-4cff-83ec-bbfbe75e0630 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:25.929222 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-ea462059-97dd-4cff-83ec-bbfbe75e0630 None None] MFA Rules not processed for user `53ea7d8704204c778d402e70dacb9bf9`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:56:25.930323 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-ea462059-97dd-4cff-83ec-bbfbe75e0630 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:56:26.004348 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1049/2085] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:56:25 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:56:26.009658 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-180604a6-5bba-4943-87e6-227ae0a09725 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:26.044696 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:56:26.084477 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-180604a6-5bba-4943-87e6-227ae0a09725 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:26.090990 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64efdc8d-ab50-415d-a51f-6cdc7f4d640d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:26.091626 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64efdc8d-ab50-415d-a51f-6cdc7f4d640d admin admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:26.091911 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64efdc8d-ab50-415d-a51f-6cdc7f4d640d admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:26.092163 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64efdc8d-ab50-415d-a51f-6cdc7f4d640d admin admin] PATH_INFO: `/v3/projects` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:26.092791 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64efdc8d-ab50-415d-a51f-6cdc7f4d640d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:26.094804 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64efdc8d-ab50-415d-a51f-6cdc7f4d640d admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:26.125522 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1037/2086] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 00:56:26 2022] POST /identity/v3/projects => generated 367 bytes in 117 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:56:26.129871 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea462059-97dd-4cff-83ec-bbfbe75e0630 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:26.142906 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea462059-97dd-4cff-83ec-bbfbe75e0630 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:26.201032 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d4c0389-e3e2-4ae2-b4ac-a96deb7a56c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:26.201537 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d4c0389-e3e2-4ae2-b4ac-a96deb7a56c8 admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:26.201817 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d4c0389-e3e2-4ae2-b4ac-a96deb7a56c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:26.202074 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d4c0389-e3e2-4ae2-b4ac-a96deb7a56c8 admin admin] PATH_INFO: `/v3/users` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:26.202694 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d4c0389-e3e2-4ae2-b4ac-a96deb7a56c8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:26.204624 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d4c0389-e3e2-4ae2-b4ac-a96deb7a56c8 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:26.228582 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1050/2087] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 00:56:26 2022] POST /identity/v3/users => generated 290 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:56:26.234263 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64efdc8d-ab50-415d-a51f-6cdc7f4d640d admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:26.250524 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64efdc8d-ab50-415d-a51f-6cdc7f4d640d admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:26.256282 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2df32fd0-c339-4f2e-9c45-65041d19985f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:26.256890 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2df32fd0-c339-4f2e-9c45-65041d19985f admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:26.257198 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2df32fd0-c339-4f2e-9c45-65041d19985f admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:26.257465 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2df32fd0-c339-4f2e-9c45-65041d19985f admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:26.258105 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2df32fd0-c339-4f2e-9c45-65041d19985f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:26.259837 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2df32fd0-c339-4f2e-9c45-65041d19985f admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:26.271661 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1038/2088] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:56:26 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 00:56:26.277737 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d4c0389-e3e2-4ae2-b4ac-a96deb7a56c8 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:26.288633 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d4c0389-e3e2-4ae2-b4ac-a96deb7a56c8 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:26.293853 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56f7d168-af50-4042-8167-13d995aaef94 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:26.294513 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f7d168-af50-4042-8167-13d995aaef94 admin admin] REQUEST_METHOD: `PUT` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:26.294773 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f7d168-af50-4042-8167-13d995aaef94 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:26.295072 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f7d168-af50-4042-8167-13d995aaef94 admin admin] PATH_INFO: `/v3/projects/94ec7a3ad563446f913b86528bedd5b6/users/cfdb787828fe41a1aa9f3e04614a9da5/roles/f08d6043261747e3b4763c744eef7dfb` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:26.312342 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56f7d168-af50-4042-8167-13d995aaef94 admin admin] RBAC: Authorizing `identity:create_grant(project_id=94ec7a3ad563446f913b86528bedd5b6, user_id=cfdb787828fe41a1aa9f3e04614a9da5, role_id=f08d6043261747e3b4763c744eef7dfb)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:26.315180 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56f7d168-af50-4042-8167-13d995aaef94 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:26.331674 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1051/2089] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 00:56:26 2022] PUT /identity/v3/projects/94ec7a3ad563446f913b86528bedd5b6/users/cfdb787828fe41a1aa9f3e04614a9da5/roles/f08d6043261747e3b4763c744eef7dfb => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 28 00:56:26.335881 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2df32fd0-c339-4f2e-9c45-65041d19985f admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:26.346490 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2df32fd0-c339-4f2e-9c45-65041d19985f admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:26.414896 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7ecbff5-a7b3-4651-80b3-62e1ad02784a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:26.415462 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7ecbff5-a7b3-4651-80b3-62e1ad02784a admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:26.415462 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7ecbff5-a7b3-4651-80b3-62e1ad02784a admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:26.415626 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7ecbff5-a7b3-4651-80b3-62e1ad02784a admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:26.416250 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7ecbff5-a7b3-4651-80b3-62e1ad02784a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:26.417888 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7ecbff5-a7b3-4651-80b3-62e1ad02784a admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:26.428504 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1039/2090] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:56:26 2022] GET /identity/v3/roles => generated 1420 bytes in 93 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 28 00:56:26.432766 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56f7d168-af50-4042-8167-13d995aaef94 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:26.443512 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56f7d168-af50-4042-8167-13d995aaef94 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:26.449291 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5dae24c-09a4-4934-8ebf-552749b0432c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:26.450036 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5dae24c-09a4-4934-8ebf-552749b0432c admin admin] REQUEST_METHOD: `PUT` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:26.450469 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5dae24c-09a4-4934-8ebf-552749b0432c admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:26.451910 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5dae24c-09a4-4934-8ebf-552749b0432c admin admin] PATH_INFO: `/v3/projects/94ec7a3ad563446f913b86528bedd5b6/users/cfdb787828fe41a1aa9f3e04614a9da5/roles/f08d6043261747e3b4763c744eef7dfb` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:26.452781 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5dae24c-09a4-4934-8ebf-552749b0432c admin admin] RBAC: Authorizing `identity:create_grant(project_id=94ec7a3ad563446f913b86528bedd5b6, user_id=cfdb787828fe41a1aa9f3e04614a9da5, role_id=f08d6043261747e3b4763c744eef7dfb)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:26.456253 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5dae24c-09a4-4934-8ebf-552749b0432c admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:26.469130 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1052/2091] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 00:56:26 2022] PUT /identity/v3/projects/94ec7a3ad563446f913b86528bedd5b6/users/cfdb787828fe41a1aa9f3e04614a9da5/roles/f08d6043261747e3b4763c744eef7dfb => generated 0 bytes in 37 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:56:26.480663 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfa62b17-4645-4679-8f29-f39094687789 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:26.480974 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfa62b17-4645-4679-8f29-f39094687789 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:26.481269 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfa62b17-4645-4679-8f29-f39094687789 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:26.511642 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-dfa62b17-4645-4679-8f29-f39094687789 None None] MFA Rules not processed for user `cfdb787828fe41a1aa9f3e04614a9da5`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:56:26.512458 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-dfa62b17-4645-4679-8f29-f39094687789 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:56:26.585995 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1040/2092] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:56:26 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:56:26.597496 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5dae24c-09a4-4934-8ebf-552749b0432c admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:26.599669 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-f5dae24c-09a4-4934-8ebf-552749b0432c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:56:26.619407 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:56:26.668291 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5dae24c-09a4-4934-8ebf-552749b0432c admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:26.672383 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-677c971d-1a34-4eea-ad45-c66be6fc5401 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:26.673084 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-677c971d-1a34-4eea-ad45-c66be6fc5401 service cinder] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:26.673362 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-677c971d-1a34-4eea-ad45-c66be6fc5401 service cinder] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:26.673630 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-677c971d-1a34-4eea-ad45-c66be6fc5401 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:26.684508 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-677c971d-1a34-4eea-ad45-c66be6fc5401 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:26.686882 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-677c971d-1a34-4eea-ad45-c66be6fc5401 service cinder] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:26.703683 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1053/2093] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:56:26 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:56:28.183264 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfa62b17-4645-4679-8f29-f39094687789 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:28.216648 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:56:28.264854 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfa62b17-4645-4679-8f29-f39094687789 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:28.269694 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f71c508-c196-4308-bdb4-319f317b12b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:28.270331 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f71c508-c196-4308-bdb4-319f317b12b3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:28.270637 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f71c508-c196-4308-bdb4-319f317b12b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:28.270889 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f71c508-c196-4308-bdb4-319f317b12b3 admin admin] PATH_INFO: `/v3/users/53ea7d8704204c778d402e70dacb9bf9` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:28.272078 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f71c508-c196-4308-bdb4-319f317b12b3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=53ea7d8704204c778d402e70dacb9bf9)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:28.274059 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f71c508-c196-4308-bdb4-319f317b12b3 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:28.325719 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-7f71c508-c196-4308-bdb4-319f317b12b3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '53ea7d8704204c778d402e70dacb9bf9'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:28.337449 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-7f71c508-c196-4308-bdb4-319f317b12b3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '53ea7d8704204c778d402e70dacb9bf9'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:28.346181 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-7f71c508-c196-4308-bdb4-319f317b12b3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '53ea7d8704204c778d402e70dacb9bf9'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:28.353343 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-7f71c508-c196-4308-bdb4-319f317b12b3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '53ea7d8704204c778d402e70dacb9bf9'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:28.354095 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-7f71c508-c196-4308-bdb4-319f317b12b3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '53ea7d8704204c778d402e70dacb9bf9'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:28.359880 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1041/2094] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 00:56:28 2022] DELETE /identity/v3/users/53ea7d8704204c778d402e70dacb9bf9 => generated 0 bytes in 178 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:56:28.364544 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-850cb440-e44f-42ac-8c54-1f1bf08e49e0 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:28.365294 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-850cb440-e44f-42ac-8c54-1f1bf08e49e0 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:28.365294 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-850cb440-e44f-42ac-8c54-1f1bf08e49e0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:28.394759 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-850cb440-e44f-42ac-8c54-1f1bf08e49e0 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:56:28.396614 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-850cb440-e44f-42ac-8c54-1f1bf08e49e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:56:28.466270 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1054/2095] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:56:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 28 00:56:28.476204 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f71c508-c196-4308-bdb4-319f317b12b3 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:28.477737 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-7f71c508-c196-4308-bdb4-319f317b12b3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:56:28.536599 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f71c508-c196-4308-bdb4-319f317b12b3 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:28.543406 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da7913d1-356e-4a0c-8285-af0eb21d1555 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:28.544289 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da7913d1-356e-4a0c-8285-af0eb21d1555 service neutron] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:28.544588 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da7913d1-356e-4a0c-8285-af0eb21d1555 service neutron] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:28.544856 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da7913d1-356e-4a0c-8285-af0eb21d1555 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:28.560314 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da7913d1-356e-4a0c-8285-af0eb21d1555 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:28.561791 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da7913d1-356e-4a0c-8285-af0eb21d1555 service neutron] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:28.572199 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1042/2096] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:56:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:56:28.877783 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-850cb440-e44f-42ac-8c54-1f1bf08e49e0 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:28.886639 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-850cb440-e44f-42ac-8c54-1f1bf08e49e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:56:28.898874 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-850cb440-e44f-42ac-8c54-1f1bf08e49e0 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:28.903529 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d89d7687-908c-47e8-a07e-4b1156f0d362 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:28.904163 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d89d7687-908c-47e8-a07e-4b1156f0d362 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:28.904439 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d89d7687-908c-47e8-a07e-4b1156f0d362 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:28.904707 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d89d7687-908c-47e8-a07e-4b1156f0d362 admin admin] PATH_INFO: `/v3/projects/d39934e430a649a7873c1010ecff4901` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:28.906023 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d89d7687-908c-47e8-a07e-4b1156f0d362 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d39934e430a649a7873c1010ecff4901)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:28.908261 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d89d7687-908c-47e8-a07e-4b1156f0d362 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:28.956765 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-d89d7687-908c-47e8-a07e-4b1156f0d362 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd39934e430a649a7873c1010ecff4901', 'request_id': 'req-d89d7687-908c-47e8-a07e-4b1156f0d362'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:28.962980 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-d89d7687-908c-47e8-a07e-4b1156f0d362 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd39934e430a649a7873c1010ecff4901', 'request_id': 'req-d89d7687-908c-47e8-a07e-4b1156f0d362'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:28.969594 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-d89d7687-908c-47e8-a07e-4b1156f0d362 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd39934e430a649a7873c1010ecff4901', 'request_id': 'req-d89d7687-908c-47e8-a07e-4b1156f0d362'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:28.982959 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-d89d7687-908c-47e8-a07e-4b1156f0d362 admin admin] The token cache is being invalidate because project d39934e430a649a7873c1010ecff4901 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81508) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 00:56:28.983176 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-d89d7687-908c-47e8-a07e-4b1156f0d362 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:28.983796 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-d89d7687-908c-47e8-a07e-4b1156f0d362 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:28.985352 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1055/2097] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 00:56:28 2022] DELETE /identity/v3/projects/d39934e430a649a7873c1010ecff4901 => generated 0 bytes in 108 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:56:28.989593 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da7913d1-356e-4a0c-8285-af0eb21d1555 service neutron] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:28.999337 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da7913d1-356e-4a0c-8285-af0eb21d1555 service neutron] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:29.043499 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:56:29.059865 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baeb9b9e-7dcb-402e-b80d-20dbf78151cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:29.060525 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baeb9b9e-7dcb-402e-b80d-20dbf78151cb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:29.060800 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baeb9b9e-7dcb-402e-b80d-20dbf78151cb admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:29.061105 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baeb9b9e-7dcb-402e-b80d-20dbf78151cb admin admin] PATH_INFO: `/v3/users/cfdb787828fe41a1aa9f3e04614a9da5` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:29.062266 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baeb9b9e-7dcb-402e-b80d-20dbf78151cb admin admin] RBAC: Authorizing `identity:delete_user(user_id=cfdb787828fe41a1aa9f3e04614a9da5)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:29.064238 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baeb9b9e-7dcb-402e-b80d-20dbf78151cb admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:29.117800 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-baeb9b9e-7dcb-402e-b80d-20dbf78151cb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cfdb787828fe41a1aa9f3e04614a9da5'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:29.134030 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-baeb9b9e-7dcb-402e-b80d-20dbf78151cb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cfdb787828fe41a1aa9f3e04614a9da5'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:29.145166 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-baeb9b9e-7dcb-402e-b80d-20dbf78151cb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cfdb787828fe41a1aa9f3e04614a9da5'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:29.153660 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-baeb9b9e-7dcb-402e-b80d-20dbf78151cb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cfdb787828fe41a1aa9f3e04614a9da5'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:29.154331 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-baeb9b9e-7dcb-402e-b80d-20dbf78151cb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cfdb787828fe41a1aa9f3e04614a9da5'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:29.160366 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1043/2098] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 00:56:28 2022] DELETE /identity/v3/users/cfdb787828fe41a1aa9f3e04614a9da5 => generated 0 bytes in 171 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:56:29.447160 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d89d7687-908c-47e8-a07e-4b1156f0d362 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:29.448933 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-d89d7687-908c-47e8-a07e-4b1156f0d362 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:56:29.466361 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d89d7687-908c-47e8-a07e-4b1156f0d362 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:29.508038 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:56:29.526909 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fb83a1d-c41a-4df0-8e56-9bc8b5201f43 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:29.527497 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb83a1d-c41a-4df0-8e56-9bc8b5201f43 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:29.527771 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb83a1d-c41a-4df0-8e56-9bc8b5201f43 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:29.528036 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb83a1d-c41a-4df0-8e56-9bc8b5201f43 admin admin] PATH_INFO: `/v3/projects/94ec7a3ad563446f913b86528bedd5b6` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:29.529287 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fb83a1d-c41a-4df0-8e56-9bc8b5201f43 admin admin] RBAC: Authorizing `identity:delete_project(project_id=94ec7a3ad563446f913b86528bedd5b6)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:29.531224 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fb83a1d-c41a-4df0-8e56-9bc8b5201f43 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:29.579657 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-2fb83a1d-c41a-4df0-8e56-9bc8b5201f43 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '94ec7a3ad563446f913b86528bedd5b6', 'request_id': 'req-2fb83a1d-c41a-4df0-8e56-9bc8b5201f43'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:29.585435 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-2fb83a1d-c41a-4df0-8e56-9bc8b5201f43 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '94ec7a3ad563446f913b86528bedd5b6', 'request_id': 'req-2fb83a1d-c41a-4df0-8e56-9bc8b5201f43'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:29.591741 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-2fb83a1d-c41a-4df0-8e56-9bc8b5201f43 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '94ec7a3ad563446f913b86528bedd5b6', 'request_id': 'req-2fb83a1d-c41a-4df0-8e56-9bc8b5201f43'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:29.606166 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-2fb83a1d-c41a-4df0-8e56-9bc8b5201f43 admin admin] The token cache is being invalidate because project 94ec7a3ad563446f913b86528bedd5b6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81508) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 00:56:29.606575 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-2fb83a1d-c41a-4df0-8e56-9bc8b5201f43 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:29.607270 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-2fb83a1d-c41a-4df0-8e56-9bc8b5201f43 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:29.608859 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1056/2099] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 00:56:29 2022] DELETE /identity/v3/projects/94ec7a3ad563446f913b86528bedd5b6 => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:56:29.613946 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6299cf33-7107-4b60-8e1b-ef0ef73a405f None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:29.614407 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6299cf33-7107-4b60-8e1b-ef0ef73a405f None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:29.614527 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6299cf33-7107-4b60-8e1b-ef0ef73a405f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:29.633934 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-6299cf33-7107-4b60-8e1b-ef0ef73a405f None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:56:29.635138 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-6299cf33-7107-4b60-8e1b-ef0ef73a405f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:56:29.706950 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1044/2100] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:56:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:56:29.717469 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e58f04cc-589b-4220-8116-c8cb31df77af None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:29.717910 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e58f04cc-589b-4220-8116-c8cb31df77af None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:29.718326 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e58f04cc-589b-4220-8116-c8cb31df77af None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:29.744728 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-e58f04cc-589b-4220-8116-c8cb31df77af None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:56:29.745985 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-e58f04cc-589b-4220-8116-c8cb31df77af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:56:29.754952 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1057/2101] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:56:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:56:29.760223 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6299cf33-7107-4b60-8e1b-ef0ef73a405f None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:29.772514 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6299cf33-7107-4b60-8e1b-ef0ef73a405f None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:29.778212 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed030f44-d753-4685-b8d2-4731a6c9dbef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:29.778767 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed030f44-d753-4685-b8d2-4731a6c9dbef admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:29.779045 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed030f44-d753-4685-b8d2-4731a6c9dbef admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:29.779316 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed030f44-d753-4685-b8d2-4731a6c9dbef admin admin] PATH_INFO: `/v3/domains` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:29.779959 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed030f44-d753-4685-b8d2-4731a6c9dbef admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:56:29.780267 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed030f44-d753-4685-b8d2-4731a6c9dbef admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:29.781910 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed030f44-d753-4685-b8d2-4731a6c9dbef admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:29.795994 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1045/2102] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Aug 28 00:56:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:56:29.801097 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e58f04cc-589b-4220-8116-c8cb31df77af None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:29.810857 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e58f04cc-589b-4220-8116-c8cb31df77af None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:29.817273 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b96460f4-6ae0-448b-b765-aa950884cacd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:29.817872 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b96460f4-6ae0-448b-b765-aa950884cacd admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:29.818146 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b96460f4-6ae0-448b-b765-aa950884cacd admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:29.818410 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b96460f4-6ae0-448b-b765-aa950884cacd admin admin] PATH_INFO: `/v3/projects` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:29.819024 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b96460f4-6ae0-448b-b765-aa950884cacd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:29.821045 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b96460f4-6ae0-448b-b765-aa950884cacd admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:29.851270 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1058/2103] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 00:56:29 2022] POST /identity/v3/projects => generated 381 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:56:29.855549 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed030f44-d753-4685-b8d2-4731a6c9dbef admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:29.866331 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed030f44-d753-4685-b8d2-4731a6c9dbef admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:29.917518 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:56:29.929106 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e16f709e-3a02-45c7-b9fc-f0236f80239d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:29.929725 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e16f709e-3a02-45c7-b9fc-f0236f80239d admin admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:29.930011 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e16f709e-3a02-45c7-b9fc-f0236f80239d admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:29.930418 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e16f709e-3a02-45c7-b9fc-f0236f80239d admin admin] PATH_INFO: `/v3/users` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:29.931089 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e16f709e-3a02-45c7-b9fc-f0236f80239d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:29.933217 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e16f709e-3a02-45c7-b9fc-f0236f80239d admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:29.962025 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1046/2104] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 00:56:29 2022] POST /identity/v3/users => generated 291 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:56:29.966477 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b96460f4-6ae0-448b-b765-aa950884cacd admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:29.978079 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b96460f4-6ae0-448b-b765-aa950884cacd admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:29.982908 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13867c5c-1b75-4cfa-9c9c-e87a4db56404 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:29.983523 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13867c5c-1b75-4cfa-9c9c-e87a4db56404 admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:29.983805 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13867c5c-1b75-4cfa-9c9c-e87a4db56404 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:29.984112 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13867c5c-1b75-4cfa-9c9c-e87a4db56404 admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:29.984709 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13867c5c-1b75-4cfa-9c9c-e87a4db56404 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:29.986407 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13867c5c-1b75-4cfa-9c9c-e87a4db56404 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:29.997687 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1059/2105] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:56:29 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 00:56:30.001889 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e16f709e-3a02-45c7-b9fc-f0236f80239d admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:30.013155 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e16f709e-3a02-45c7-b9fc-f0236f80239d admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:30.018247 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-509d13c0-bc99-441f-b9e8-cee08c4229ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:30.018843 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-509d13c0-bc99-441f-b9e8-cee08c4229ef admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:30.019140 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-509d13c0-bc99-441f-b9e8-cee08c4229ef admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:30.019440 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-509d13c0-bc99-441f-b9e8-cee08c4229ef admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:30.020081 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-509d13c0-bc99-441f-b9e8-cee08c4229ef admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:30.021834 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-509d13c0-bc99-441f-b9e8-cee08c4229ef admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:30.033168 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1047/2106] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:56:30 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 00:56:30.037292 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13867c5c-1b75-4cfa-9c9c-e87a4db56404 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:30.047893 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13867c5c-1b75-4cfa-9c9c-e87a4db56404 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:30.055986 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9f93dc6-782e-4896-97ae-c79ae2aefe07 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:30.056678 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9f93dc6-782e-4896-97ae-c79ae2aefe07 admin admin] REQUEST_METHOD: `PUT` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:30.057133 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9f93dc6-782e-4896-97ae-c79ae2aefe07 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:30.057409 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9f93dc6-782e-4896-97ae-c79ae2aefe07 admin admin] PATH_INFO: `/v3/projects/b281a29283e64a8a89fd5a7558c336f8/users/972b867a6bae4e9a8c69d20ad818ba01/roles/0487ef7e943e4701915cb8ce0f2899d7` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:30.075312 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9f93dc6-782e-4896-97ae-c79ae2aefe07 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b281a29283e64a8a89fd5a7558c336f8, user_id=972b867a6bae4e9a8c69d20ad818ba01, role_id=0487ef7e943e4701915cb8ce0f2899d7)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:30.078019 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9f93dc6-782e-4896-97ae-c79ae2aefe07 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:30.092355 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1060/2107] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 00:56:30 2022] PUT /identity/v3/projects/b281a29283e64a8a89fd5a7558c336f8/users/972b867a6bae4e9a8c69d20ad818ba01/roles/0487ef7e943e4701915cb8ce0f2899d7 => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:56:30.103863 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b7ef4e-787f-4059-afc1-039b4a0f6dc0 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:30.104155 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b7ef4e-787f-4059-afc1-039b4a0f6dc0 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:30.104447 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b7ef4e-787f-4059-afc1-039b4a0f6dc0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:30.134641 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-e8b7ef4e-787f-4059-afc1-039b4a0f6dc0 None None] MFA Rules not processed for user `972b867a6bae4e9a8c69d20ad818ba01`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:56:30.135865 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-e8b7ef4e-787f-4059-afc1-039b4a0f6dc0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:56:30.204902 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1048/2108] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:56:30 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 28 00:56:30.211086 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9f93dc6-782e-4896-97ae-c79ae2aefe07 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:30.272501 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9f93dc6-782e-4896-97ae-c79ae2aefe07 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:30.277854 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35a8931a-d9aa-4aa8-940c-680e278f9955 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:30.278457 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a8931a-d9aa-4aa8-940c-680e278f9955 admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:30.278735 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a8931a-d9aa-4aa8-940c-680e278f9955 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:30.279007 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a8931a-d9aa-4aa8-940c-680e278f9955 admin admin] PATH_INFO: `/v3/projects` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:30.279638 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35a8931a-d9aa-4aa8-940c-680e278f9955 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:30.281638 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35a8931a-d9aa-4aa8-940c-680e278f9955 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:30.297717 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:56:30.315461 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1061/2109] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 00:56:30 2022] POST /identity/v3/projects => generated 381 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:56:30.317663 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8b7ef4e-787f-4059-afc1-039b4a0f6dc0 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:30.329849 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8b7ef4e-787f-4059-afc1-039b4a0f6dc0 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:30.390653 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6f0d1fe-fda1-4345-9d1a-f39b323b1efd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:30.391315 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f0d1fe-fda1-4345-9d1a-f39b323b1efd admin admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:30.391599 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f0d1fe-fda1-4345-9d1a-f39b323b1efd admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:30.391870 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f0d1fe-fda1-4345-9d1a-f39b323b1efd admin admin] PATH_INFO: `/v3/users` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:30.392514 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6f0d1fe-fda1-4345-9d1a-f39b323b1efd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:30.394581 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6f0d1fe-fda1-4345-9d1a-f39b323b1efd admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:30.416364 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1049/2110] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 00:56:30 2022] POST /identity/v3/users => generated 297 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:56:30.420369 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35a8931a-d9aa-4aa8-940c-680e278f9955 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:30.431520 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35a8931a-d9aa-4aa8-940c-680e278f9955 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:30.437306 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b607617f-42ba-49d5-ae17-f2c9d7b6d0ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:30.437937 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b607617f-42ba-49d5-ae17-f2c9d7b6d0ed admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:30.438569 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b607617f-42ba-49d5-ae17-f2c9d7b6d0ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:30.438837 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b607617f-42ba-49d5-ae17-f2c9d7b6d0ed admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:30.439477 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b607617f-42ba-49d5-ae17-f2c9d7b6d0ed admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:30.441111 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b607617f-42ba-49d5-ae17-f2c9d7b6d0ed admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:30.453395 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1062/2111] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:56:30 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 00:56:30.458174 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6f0d1fe-fda1-4345-9d1a-f39b323b1efd admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:30.473509 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6f0d1fe-fda1-4345-9d1a-f39b323b1efd admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:30.482022 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-688df672-4839-426c-aaa3-8ac19df14394 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:30.482973 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-688df672-4839-426c-aaa3-8ac19df14394 admin admin] REQUEST_METHOD: `PUT` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:30.483334 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-688df672-4839-426c-aaa3-8ac19df14394 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:30.483710 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-688df672-4839-426c-aaa3-8ac19df14394 admin admin] PATH_INFO: `/v3/projects/5738a5d663fd40d186039018d1af6718/users/9cb89f8b4c224014ae83c6a334c2f94b/roles/f08d6043261747e3b4763c744eef7dfb` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:30.501826 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-688df672-4839-426c-aaa3-8ac19df14394 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5738a5d663fd40d186039018d1af6718, user_id=9cb89f8b4c224014ae83c6a334c2f94b, role_id=f08d6043261747e3b4763c744eef7dfb)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:30.504540 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-688df672-4839-426c-aaa3-8ac19df14394 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:30.520783 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1050/2112] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 00:56:30 2022] PUT /identity/v3/projects/5738a5d663fd40d186039018d1af6718/users/9cb89f8b4c224014ae83c6a334c2f94b/roles/f08d6043261747e3b4763c744eef7dfb => generated 0 bytes in 63 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:56:30.525212 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b607617f-42ba-49d5-ae17-f2c9d7b6d0ed admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:30.536354 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b607617f-42ba-49d5-ae17-f2c9d7b6d0ed admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:30.598339 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f6dfc07-cc54-44ab-b420-0626fba1c478 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:30.598866 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f6dfc07-cc54-44ab-b420-0626fba1c478 admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:30.599141 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f6dfc07-cc54-44ab-b420-0626fba1c478 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:30.599407 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f6dfc07-cc54-44ab-b420-0626fba1c478 admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:30.600039 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f6dfc07-cc54-44ab-b420-0626fba1c478 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:30.601669 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f6dfc07-cc54-44ab-b420-0626fba1c478 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:30.612694 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1063/2113] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:56:30 2022] GET /identity/v3/roles => generated 1420 bytes in 88 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 00:56:30.617856 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-688df672-4839-426c-aaa3-8ac19df14394 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:30.629226 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-688df672-4839-426c-aaa3-8ac19df14394 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:30.634873 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a9f1139-d2fc-4067-910a-6d2df50c279e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:30.635614 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a9f1139-d2fc-4067-910a-6d2df50c279e admin admin] REQUEST_METHOD: `PUT` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:30.635891 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a9f1139-d2fc-4067-910a-6d2df50c279e admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:30.636160 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a9f1139-d2fc-4067-910a-6d2df50c279e admin admin] PATH_INFO: `/v3/projects/5738a5d663fd40d186039018d1af6718/users/9cb89f8b4c224014ae83c6a334c2f94b/roles/f08d6043261747e3b4763c744eef7dfb` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:30.638204 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a9f1139-d2fc-4067-910a-6d2df50c279e admin admin] RBAC: Authorizing `identity:create_grant(project_id=5738a5d663fd40d186039018d1af6718, user_id=9cb89f8b4c224014ae83c6a334c2f94b, role_id=f08d6043261747e3b4763c744eef7dfb)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:30.640875 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a9f1139-d2fc-4067-910a-6d2df50c279e admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:30.652031 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1051/2114] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 00:56:30 2022] PUT /identity/v3/projects/5738a5d663fd40d186039018d1af6718/users/9cb89f8b4c224014ae83c6a334c2f94b/roles/f08d6043261747e3b4763c744eef7dfb => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:56:30.662620 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8af2e335-36c5-4826-b056-64ad730ff631 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:30.662900 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8af2e335-36c5-4826-b056-64ad730ff631 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:30.663178 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8af2e335-36c5-4826-b056-64ad730ff631 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:30.691686 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-8af2e335-36c5-4826-b056-64ad730ff631 None None] MFA Rules not processed for user `9cb89f8b4c224014ae83c6a334c2f94b`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:56:30.692908 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-8af2e335-36c5-4826-b056-64ad730ff631 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:56:30.766531 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1064/2115] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:56:30 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:56:30.777698 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a9f1139-d2fc-4067-910a-6d2df50c279e admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:30.779743 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-9a9f1139-d2fc-4067-910a-6d2df50c279e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:56:30.794989 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:56:30.849948 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a9f1139-d2fc-4067-910a-6d2df50c279e admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:30.853723 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ba69199-ba57-41fb-b6be-69db7c6b3045 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:30.854378 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ba69199-ba57-41fb-b6be-69db7c6b3045 service cinder] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:30.854605 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ba69199-ba57-41fb-b6be-69db7c6b3045 service cinder] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:30.854875 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ba69199-ba57-41fb-b6be-69db7c6b3045 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:30.864096 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ba69199-ba57-41fb-b6be-69db7c6b3045 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:30.865647 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ba69199-ba57-41fb-b6be-69db7c6b3045 service cinder] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:30.875357 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1052/2116] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:56:30 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:56:31.329481 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8af2e335-36c5-4826-b056-64ad730ff631 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:31.404527 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8af2e335-36c5-4826-b056-64ad730ff631 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:31.409547 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08e140d5-0b2e-4d6c-b132-94fc59c061e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:31.410181 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08e140d5-0b2e-4d6c-b132-94fc59c061e3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:31.410459 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08e140d5-0b2e-4d6c-b132-94fc59c061e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:31.410729 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08e140d5-0b2e-4d6c-b132-94fc59c061e3 admin admin] PATH_INFO: `/v3/users/972b867a6bae4e9a8c69d20ad818ba01` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:31.411885 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08e140d5-0b2e-4d6c-b132-94fc59c061e3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=972b867a6bae4e9a8c69d20ad818ba01)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:31.413879 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08e140d5-0b2e-4d6c-b132-94fc59c061e3 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:31.419278 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:INSERT=1,keystone:SELECT=99 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:56:31.467363 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-08e140d5-0b2e-4d6c-b132-94fc59c061e3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '972b867a6bae4e9a8c69d20ad818ba01'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:31.468039 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-08e140d5-0b2e-4d6c-b132-94fc59c061e3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '972b867a6bae4e9a8c69d20ad818ba01'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:31.476153 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-08e140d5-0b2e-4d6c-b132-94fc59c061e3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '972b867a6bae4e9a8c69d20ad818ba01'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:31.483065 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-08e140d5-0b2e-4d6c-b132-94fc59c061e3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '972b867a6bae4e9a8c69d20ad818ba01'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:31.483700 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-08e140d5-0b2e-4d6c-b132-94fc59c061e3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '972b867a6bae4e9a8c69d20ad818ba01'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:31.502229 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1065/2117] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 00:56:31 2022] DELETE /identity/v3/users/972b867a6bae4e9a8c69d20ad818ba01 => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:56:31.507120 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c7e0da5-8550-4d43-8bbd-7091776f3a95 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:31.507434 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c7e0da5-8550-4d43-8bbd-7091776f3a95 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:31.507711 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c7e0da5-8550-4d43-8bbd-7091776f3a95 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:31.528183 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-2c7e0da5-8550-4d43-8bbd-7091776f3a95 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:56:31.529446 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-2c7e0da5-8550-4d43-8bbd-7091776f3a95 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:56:31.599788 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1053/2118] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:56:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:56:31.610208 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08e140d5-0b2e-4d6c-b132-94fc59c061e3 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:31.612344 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-08e140d5-0b2e-4d6c-b132-94fc59c061e3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:56:31.677153 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08e140d5-0b2e-4d6c-b132-94fc59c061e3 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:31.681167 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb44923b-47e5-4ccb-b428-a2aded3ecde1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:31.681782 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb44923b-47e5-4ccb-b428-a2aded3ecde1 service neutron] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:31.682064 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb44923b-47e5-4ccb-b428-a2aded3ecde1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:31.682338 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb44923b-47e5-4ccb-b428-a2aded3ecde1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:31.693149 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb44923b-47e5-4ccb-b428-a2aded3ecde1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:31.694597 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb44923b-47e5-4ccb-b428-a2aded3ecde1 service neutron] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:31.705255 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1066/2119] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:56:31 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:56:31.995352 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c7e0da5-8550-4d43-8bbd-7091776f3a95 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:31.996816 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-2c7e0da5-8550-4d43-8bbd-7091776f3a95 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:56:32.012767 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c7e0da5-8550-4d43-8bbd-7091776f3a95 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:32.020157 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28d9edea-0df5-4d04-8040-6b25374bc783 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:32.021217 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d9edea-0df5-4d04-8040-6b25374bc783 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:32.021684 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d9edea-0df5-4d04-8040-6b25374bc783 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:32.022136 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d9edea-0df5-4d04-8040-6b25374bc783 admin admin] PATH_INFO: `/v3/projects/b281a29283e64a8a89fd5a7558c336f8` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:32.023958 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28d9edea-0df5-4d04-8040-6b25374bc783 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b281a29283e64a8a89fd5a7558c336f8)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:32.027054 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28d9edea-0df5-4d04-8040-6b25374bc783 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:32.095865 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-28d9edea-0df5-4d04-8040-6b25374bc783 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b281a29283e64a8a89fd5a7558c336f8', 'request_id': 'req-28d9edea-0df5-4d04-8040-6b25374bc783'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:32.104651 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-28d9edea-0df5-4d04-8040-6b25374bc783 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b281a29283e64a8a89fd5a7558c336f8', 'request_id': 'req-28d9edea-0df5-4d04-8040-6b25374bc783'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:32.112828 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-28d9edea-0df5-4d04-8040-6b25374bc783 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b281a29283e64a8a89fd5a7558c336f8', 'request_id': 'req-28d9edea-0df5-4d04-8040-6b25374bc783'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:32.135698 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-28d9edea-0df5-4d04-8040-6b25374bc783 admin admin] The token cache is being invalidate because project b281a29283e64a8a89fd5a7558c336f8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81507) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 00:56:32.135966 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-28d9edea-0df5-4d04-8040-6b25374bc783 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:32.136724 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-28d9edea-0df5-4d04-8040-6b25374bc783 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:32.138303 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1054/2120] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 00:56:31 2022] DELETE /identity/v3/projects/b281a29283e64a8a89fd5a7558c336f8 => generated 0 bytes in 144 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 28 00:56:32.142467 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb44923b-47e5-4ccb-b428-a2aded3ecde1 service neutron] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:32.152580 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb44923b-47e5-4ccb-b428-a2aded3ecde1 service neutron] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:32.214532 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05a86970-8799-483f-b1f4-94eeb5b043b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:32.215171 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05a86970-8799-483f-b1f4-94eeb5b043b4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:32.215448 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05a86970-8799-483f-b1f4-94eeb5b043b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:32.215716 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05a86970-8799-483f-b1f4-94eeb5b043b4 admin admin] PATH_INFO: `/v3/users/9cb89f8b4c224014ae83c6a334c2f94b` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:32.216848 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05a86970-8799-483f-b1f4-94eeb5b043b4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9cb89f8b4c224014ae83c6a334c2f94b)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:32.218797 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05a86970-8799-483f-b1f4-94eeb5b043b4 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:32.250719 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=84,keystone:DELETE=15,keystone:INSERT=1 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:56:32.278621 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-05a86970-8799-483f-b1f4-94eeb5b043b4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9cb89f8b4c224014ae83c6a334c2f94b'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:32.279910 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-05a86970-8799-483f-b1f4-94eeb5b043b4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9cb89f8b4c224014ae83c6a334c2f94b'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:32.291437 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-05a86970-8799-483f-b1f4-94eeb5b043b4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9cb89f8b4c224014ae83c6a334c2f94b'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:32.300218 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-05a86970-8799-483f-b1f4-94eeb5b043b4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9cb89f8b4c224014ae83c6a334c2f94b'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:32.301085 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-05a86970-8799-483f-b1f4-94eeb5b043b4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9cb89f8b4c224014ae83c6a334c2f94b'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:32.323054 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1067/2121] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 00:56:32 2022] DELETE /identity/v3/users/9cb89f8b4c224014ae83c6a334c2f94b => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:56:32.670046 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28d9edea-0df5-4d04-8040-6b25374bc783 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:32.671532 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-28d9edea-0df5-4d04-8040-6b25374bc783 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:56:32.685818 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28d9edea-0df5-4d04-8040-6b25374bc783 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:32.730933 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:56:32.757362 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b76bc50a-b9ab-4d0c-92f5-62fb43c0ca3d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:32.757993 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b76bc50a-b9ab-4d0c-92f5-62fb43c0ca3d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:32.758282 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b76bc50a-b9ab-4d0c-92f5-62fb43c0ca3d admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:32.758549 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b76bc50a-b9ab-4d0c-92f5-62fb43c0ca3d admin admin] PATH_INFO: `/v3/projects/5738a5d663fd40d186039018d1af6718` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:32.759701 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b76bc50a-b9ab-4d0c-92f5-62fb43c0ca3d admin admin] RBAC: Authorizing `identity:delete_project(project_id=5738a5d663fd40d186039018d1af6718)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:32.761732 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b76bc50a-b9ab-4d0c-92f5-62fb43c0ca3d admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:32.811061 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-b76bc50a-b9ab-4d0c-92f5-62fb43c0ca3d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5738a5d663fd40d186039018d1af6718', 'request_id': 'req-b76bc50a-b9ab-4d0c-92f5-62fb43c0ca3d'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:32.816915 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-b76bc50a-b9ab-4d0c-92f5-62fb43c0ca3d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5738a5d663fd40d186039018d1af6718', 'request_id': 'req-b76bc50a-b9ab-4d0c-92f5-62fb43c0ca3d'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:32.823298 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-b76bc50a-b9ab-4d0c-92f5-62fb43c0ca3d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5738a5d663fd40d186039018d1af6718', 'request_id': 'req-b76bc50a-b9ab-4d0c-92f5-62fb43c0ca3d'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:32.841776 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-b76bc50a-b9ab-4d0c-92f5-62fb43c0ca3d admin admin] The token cache is being invalidate because project 5738a5d663fd40d186039018d1af6718 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81507) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 00:56:32.842070 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-b76bc50a-b9ab-4d0c-92f5-62fb43c0ca3d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:32.842855 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-b76bc50a-b9ab-4d0c-92f5-62fb43c0ca3d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:32.844628 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1055/2122] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 00:56:32 2022] DELETE /identity/v3/projects/5738a5d663fd40d186039018d1af6718 => generated 0 bytes in 175 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:56:32.850899 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5c06b7e-ee73-42f9-a3cf-9f260a46f9c5 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:32.851814 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5c06b7e-ee73-42f9-a3cf-9f260a46f9c5 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:32.851814 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5c06b7e-ee73-42f9-a3cf-9f260a46f9c5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:32.879684 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-e5c06b7e-ee73-42f9-a3cf-9f260a46f9c5 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:56:32.880914 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-e5c06b7e-ee73-42f9-a3cf-9f260a46f9c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:56:32.943475 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1068/2123] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:56:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:56:32.955630 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19699fa2-8467-4ae5-b97a-8e5f98a4d3bb None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:32.955869 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19699fa2-8467-4ae5-b97a-8e5f98a4d3bb None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:32.956049 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19699fa2-8467-4ae5-b97a-8e5f98a4d3bb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:32.976197 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-19699fa2-8467-4ae5-b97a-8e5f98a4d3bb None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:56:32.977441 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-19699fa2-8467-4ae5-b97a-8e5f98a4d3bb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:56:32.986235 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1056/2124] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:56:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:56:32.990577 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5c06b7e-ee73-42f9-a3cf-9f260a46f9c5 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:33.000752 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5c06b7e-ee73-42f9-a3cf-9f260a46f9c5 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:33.005767 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d45a4d8e-0ffb-40fe-b6b2-3abeffc283f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:33.006412 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d45a4d8e-0ffb-40fe-b6b2-3abeffc283f1 admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:33.006679 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d45a4d8e-0ffb-40fe-b6b2-3abeffc283f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:33.006988 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d45a4d8e-0ffb-40fe-b6b2-3abeffc283f1 admin admin] PATH_INFO: `/v3/domains` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:33.007631 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d45a4d8e-0ffb-40fe-b6b2-3abeffc283f1 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:56:33.007971 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d45a4d8e-0ffb-40fe-b6b2-3abeffc283f1 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:33.009683 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d45a4d8e-0ffb-40fe-b6b2-3abeffc283f1 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:33.021961 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1069/2125] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Aug 28 00:56:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:56:33.028354 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19699fa2-8467-4ae5-b97a-8e5f98a4d3bb None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:33.040898 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19699fa2-8467-4ae5-b97a-8e5f98a4d3bb None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:33.052381 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0de5f254-3d0b-402a-aebf-1f4a9886727d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:33.053916 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0de5f254-3d0b-402a-aebf-1f4a9886727d admin admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:33.054242 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0de5f254-3d0b-402a-aebf-1f4a9886727d admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:33.054514 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0de5f254-3d0b-402a-aebf-1f4a9886727d admin admin] PATH_INFO: `/v3/projects` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:33.055161 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0de5f254-3d0b-402a-aebf-1f4a9886727d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:33.057101 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0de5f254-3d0b-402a-aebf-1f4a9886727d admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:33.090082 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1057/2126] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 00:56:33 2022] POST /identity/v3/projects => generated 371 bytes in 63 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:56:33.095801 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d45a4d8e-0ffb-40fe-b6b2-3abeffc283f1 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:33.106417 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d45a4d8e-0ffb-40fe-b6b2-3abeffc283f1 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:33.170255 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0f4b094-c4da-4025-81b6-9e4351bc9360 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:33.170861 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0f4b094-c4da-4025-81b6-9e4351bc9360 admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:33.171141 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0f4b094-c4da-4025-81b6-9e4351bc9360 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:33.171414 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0f4b094-c4da-4025-81b6-9e4351bc9360 admin admin] PATH_INFO: `/v3/users` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:33.172038 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0f4b094-c4da-4025-81b6-9e4351bc9360 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:33.174069 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0f4b094-c4da-4025-81b6-9e4351bc9360 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:33.197167 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1070/2127] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 00:56:33 2022] POST /identity/v3/users => generated 286 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:56:33.202897 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0de5f254-3d0b-402a-aebf-1f4a9886727d admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:33.214503 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0de5f254-3d0b-402a-aebf-1f4a9886727d admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:33.219880 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b5eb215-f77a-48ee-a6d2-bf0a29e00509 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:33.220478 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b5eb215-f77a-48ee-a6d2-bf0a29e00509 admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:33.220751 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b5eb215-f77a-48ee-a6d2-bf0a29e00509 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:33.221064 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b5eb215-f77a-48ee-a6d2-bf0a29e00509 admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:33.221701 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b5eb215-f77a-48ee-a6d2-bf0a29e00509 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:33.223311 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b5eb215-f77a-48ee-a6d2-bf0a29e00509 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:33.234223 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1058/2128] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:56:33 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 00:56:33.238920 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0f4b094-c4da-4025-81b6-9e4351bc9360 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:33.250330 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0f4b094-c4da-4025-81b6-9e4351bc9360 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:33.256463 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb7693ec-1279-4387-aa7a-0bc525336b6d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:33.257108 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb7693ec-1279-4387-aa7a-0bc525336b6d admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:33.257384 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb7693ec-1279-4387-aa7a-0bc525336b6d admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:33.257648 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb7693ec-1279-4387-aa7a-0bc525336b6d admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:33.258281 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb7693ec-1279-4387-aa7a-0bc525336b6d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:33.260003 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb7693ec-1279-4387-aa7a-0bc525336b6d admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:33.273116 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1071/2129] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:56:33 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 00:56:33.277555 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b5eb215-f77a-48ee-a6d2-bf0a29e00509 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:33.291018 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b5eb215-f77a-48ee-a6d2-bf0a29e00509 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:33.296294 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af1e8609-2289-4459-895a-7f1f114f79f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:33.297039 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af1e8609-2289-4459-895a-7f1f114f79f5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:33.297351 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af1e8609-2289-4459-895a-7f1f114f79f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:33.297625 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af1e8609-2289-4459-895a-7f1f114f79f5 admin admin] PATH_INFO: `/v3/projects/bc3d0dc67a6f44c59cb2007ffe156dec/users/69aa1d53c9fa4a9a942d18c15886c93f/roles/0487ef7e943e4701915cb8ce0f2899d7` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:33.313198 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af1e8609-2289-4459-895a-7f1f114f79f5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bc3d0dc67a6f44c59cb2007ffe156dec, user_id=69aa1d53c9fa4a9a942d18c15886c93f, role_id=0487ef7e943e4701915cb8ce0f2899d7)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:33.315848 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af1e8609-2289-4459-895a-7f1f114f79f5 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:33.331402 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1059/2130] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 00:56:33 2022] PUT /identity/v3/projects/bc3d0dc67a6f44c59cb2007ffe156dec/users/69aa1d53c9fa4a9a942d18c15886c93f/roles/0487ef7e943e4701915cb8ce0f2899d7 => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:56:33.336631 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f879295b-3ce6-409f-b178-654cfd2a1cab None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:33.336919 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f879295b-3ce6-409f-b178-654cfd2a1cab None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:33.337220 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f879295b-3ce6-409f-b178-654cfd2a1cab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:33.357028 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-f879295b-3ce6-409f-b178-654cfd2a1cab None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:56:33.358268 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-f879295b-3ce6-409f-b178-654cfd2a1cab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:56:33.363961 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=91,keystone:DELETE=5,keystone:INSERT=4 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:56:33.424427 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1072/2131] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:56:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:56:33.434259 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af1e8609-2289-4459-895a-7f1f114f79f5 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:33.436141 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-af1e8609-2289-4459-895a-7f1f114f79f5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:56:33.499116 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af1e8609-2289-4459-895a-7f1f114f79f5 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:33.503480 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fb646d6-526c-4280-86ce-b43e738b7840 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:33.504095 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb646d6-526c-4280-86ce-b43e738b7840 service neutron] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:33.504380 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb646d6-526c-4280-86ce-b43e738b7840 service neutron] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:33.504632 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb646d6-526c-4280-86ce-b43e738b7840 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:33.515251 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fb646d6-526c-4280-86ce-b43e738b7840 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:33.516895 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fb646d6-526c-4280-86ce-b43e738b7840 service neutron] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:33.527033 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1060/2132] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:56:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:56:34.280413 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7ef7b9f-cd1b-4e91-89df-b0030df90c20 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:34.280995 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7ef7b9f-cd1b-4e91-89df-b0030df90c20 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:34.281109 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7ef7b9f-cd1b-4e91-89df-b0030df90c20 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:34.313458 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-c7ef7b9f-cd1b-4e91-89df-b0030df90c20 None None] MFA Rules not processed for user `69aa1d53c9fa4a9a942d18c15886c93f`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:56:34.314946 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-c7ef7b9f-cd1b-4e91-89df-b0030df90c20 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:56:34.398931 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1073/2133] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:56:34 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:56:34.403798 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fb646d6-526c-4280-86ce-b43e738b7840 service neutron] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:34.414098 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fb646d6-526c-4280-86ce-b43e738b7840 service neutron] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:34.418998 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4db822b-c1b6-4afd-ba98-8d69dfae1948 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:34.419604 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4db822b-c1b6-4afd-ba98-8d69dfae1948 admin admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:34.419933 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4db822b-c1b6-4afd-ba98-8d69dfae1948 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:34.420159 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4db822b-c1b6-4afd-ba98-8d69dfae1948 admin admin] PATH_INFO: `/v3/projects` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:34.420818 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4db822b-c1b6-4afd-ba98-8d69dfae1948 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:34.422882 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4db822b-c1b6-4afd-ba98-8d69dfae1948 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:34.428020 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:56:34.459016 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1061/2134] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 00:56:34 2022] POST /identity/v3/projects => generated 369 bytes in 56 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:56:34.463362 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7ef7b9f-cd1b-4e91-89df-b0030df90c20 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:34.474999 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7ef7b9f-cd1b-4e91-89df-b0030df90c20 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:34.533706 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a7078ef-7727-486d-8fb5-6516aa91a01b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:34.534318 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a7078ef-7727-486d-8fb5-6516aa91a01b admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:34.534591 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a7078ef-7727-486d-8fb5-6516aa91a01b admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:34.534859 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a7078ef-7727-486d-8fb5-6516aa91a01b admin admin] PATH_INFO: `/v3/users` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:34.535529 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a7078ef-7727-486d-8fb5-6516aa91a01b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:34.537484 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a7078ef-7727-486d-8fb5-6516aa91a01b admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:34.559989 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1074/2135] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 00:56:34 2022] POST /identity/v3/users => generated 291 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:56:34.564529 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4db822b-c1b6-4afd-ba98-8d69dfae1948 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:34.575519 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4db822b-c1b6-4afd-ba98-8d69dfae1948 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:34.580834 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b4333e2-f541-4cf7-b684-7832cfe69688 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:34.581465 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b4333e2-f541-4cf7-b684-7832cfe69688 admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:34.581746 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b4333e2-f541-4cf7-b684-7832cfe69688 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:34.582015 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b4333e2-f541-4cf7-b684-7832cfe69688 admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:34.582657 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b4333e2-f541-4cf7-b684-7832cfe69688 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:34.584269 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b4333e2-f541-4cf7-b684-7832cfe69688 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:34.594665 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1062/2136] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:56:34 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 00:56:34.599584 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a7078ef-7727-486d-8fb5-6516aa91a01b admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:34.610162 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a7078ef-7727-486d-8fb5-6516aa91a01b admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:34.617475 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-697125b3-1135-4ca4-9ffa-ab15fdb9d094 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:34.618146 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-697125b3-1135-4ca4-9ffa-ab15fdb9d094 admin admin] REQUEST_METHOD: `PUT` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:34.618410 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-697125b3-1135-4ca4-9ffa-ab15fdb9d094 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:34.618695 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-697125b3-1135-4ca4-9ffa-ab15fdb9d094 admin admin] PATH_INFO: `/v3/projects/d5263ed3d2ee43f4b99d605d054fa20e/users/080d9f9f889d457fb08929c3e1db4109/roles/f08d6043261747e3b4763c744eef7dfb` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:34.635046 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-697125b3-1135-4ca4-9ffa-ab15fdb9d094 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d5263ed3d2ee43f4b99d605d054fa20e, user_id=080d9f9f889d457fb08929c3e1db4109, role_id=f08d6043261747e3b4763c744eef7dfb)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:34.637819 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-697125b3-1135-4ca4-9ffa-ab15fdb9d094 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:34.652539 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1075/2137] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 00:56:34 2022] PUT /identity/v3/projects/d5263ed3d2ee43f4b99d605d054fa20e/users/080d9f9f889d457fb08929c3e1db4109/roles/f08d6043261747e3b4763c744eef7dfb => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 28 00:56:34.657041 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b4333e2-f541-4cf7-b684-7832cfe69688 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:34.667106 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b4333e2-f541-4cf7-b684-7832cfe69688 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:34.732217 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdc85f04-ece2-4b34-b6be-fee6754dad6c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:34.732851 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdc85f04-ece2-4b34-b6be-fee6754dad6c admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:34.733167 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdc85f04-ece2-4b34-b6be-fee6754dad6c admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:34.733442 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdc85f04-ece2-4b34-b6be-fee6754dad6c admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:34.734078 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdc85f04-ece2-4b34-b6be-fee6754dad6c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:34.735797 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdc85f04-ece2-4b34-b6be-fee6754dad6c admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:34.747047 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1063/2138] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:56:34 2022] GET /identity/v3/roles => generated 1420 bytes in 91 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 28 00:56:34.751354 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-697125b3-1135-4ca4-9ffa-ab15fdb9d094 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:34.762278 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-697125b3-1135-4ca4-9ffa-ab15fdb9d094 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:34.767076 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d7f0a2c-9bf7-4348-9ef3-dcd4bea49adc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:34.767758 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d7f0a2c-9bf7-4348-9ef3-dcd4bea49adc admin admin] REQUEST_METHOD: `PUT` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:34.768056 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d7f0a2c-9bf7-4348-9ef3-dcd4bea49adc admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:34.768340 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d7f0a2c-9bf7-4348-9ef3-dcd4bea49adc admin admin] PATH_INFO: `/v3/projects/d5263ed3d2ee43f4b99d605d054fa20e/users/080d9f9f889d457fb08929c3e1db4109/roles/f08d6043261747e3b4763c744eef7dfb` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:34.770303 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d7f0a2c-9bf7-4348-9ef3-dcd4bea49adc admin admin] RBAC: Authorizing `identity:create_grant(project_id=d5263ed3d2ee43f4b99d605d054fa20e, user_id=080d9f9f889d457fb08929c3e1db4109, role_id=f08d6043261747e3b4763c744eef7dfb)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:34.772993 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d7f0a2c-9bf7-4348-9ef3-dcd4bea49adc admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:34.775471 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:56:34.787103 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1076/2139] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 00:56:34 2022] PUT /identity/v3/projects/d5263ed3d2ee43f4b99d605d054fa20e/users/080d9f9f889d457fb08929c3e1db4109/roles/f08d6043261747e3b4763c744eef7dfb => generated 0 bytes in 36 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:56:35.466699 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b74c0162-8a6e-4816-8fe2-759035caeba2 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:35.467269 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b74c0162-8a6e-4816-8fe2-759035caeba2 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:35.467367 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b74c0162-8a6e-4816-8fe2-759035caeba2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:35.498231 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-b74c0162-8a6e-4816-8fe2-759035caeba2 None None] MFA Rules not processed for user `080d9f9f889d457fb08929c3e1db4109`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:56:35.499465 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-b74c0162-8a6e-4816-8fe2-759035caeba2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:56:35.576274 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1064/2140] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:56:35 2022] POST /identity/v3/auth/tokens => generated 3313 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:56:35.588614 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d7f0a2c-9bf7-4348-9ef3-dcd4bea49adc admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:35.590524 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-3d7f0a2c-9bf7-4348-9ef3-dcd4bea49adc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:56:35.653092 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d7f0a2c-9bf7-4348-9ef3-dcd4bea49adc admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:35.657784 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afe8a1d8-c118-4588-b560-92c02991c7d9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:35.658424 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afe8a1d8-c118-4588-b560-92c02991c7d9 service cinder] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:35.659651 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afe8a1d8-c118-4588-b560-92c02991c7d9 service cinder] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:35.659651 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afe8a1d8-c118-4588-b560-92c02991c7d9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:35.722519 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afe8a1d8-c118-4588-b560-92c02991c7d9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:35.724030 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afe8a1d8-c118-4588-b560-92c02991c7d9 service cinder] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:35.733913 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1077/2141] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:56:35 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 146 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:56:40.324992 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b74c0162-8a6e-4816-8fe2-759035caeba2 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:40.336581 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b74c0162-8a6e-4816-8fe2-759035caeba2 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:40.341363 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04152784-ece1-4d49-946f-149de9a78404 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:40.341963 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04152784-ece1-4d49-946f-149de9a78404 service cinder] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:40.342304 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04152784-ece1-4d49-946f-149de9a78404 service cinder] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:40.342576 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04152784-ece1-4d49-946f-149de9a78404 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:40.352547 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04152784-ece1-4d49-946f-149de9a78404 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:40.354069 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04152784-ece1-4d49-946f-149de9a78404 service cinder] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:40.364397 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1065/2142] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:56:40 2022] GET /identity/v3/auth/tokens => generated 3313 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:56:45.311513 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afe8a1d8-c118-4588-b560-92c02991c7d9 service cinder] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:45.314156 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-afe8a1d8-c118-4588-b560-92c02991c7d9 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:56:45.416879 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afe8a1d8-c118-4588-b560-92c02991c7d9 service cinder] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:45.425736 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32095899-6f35-40bd-912c-3a431d81ecfa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:45.426822 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32095899-6f35-40bd-912c-3a431d81ecfa admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:45.427314 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32095899-6f35-40bd-912c-3a431d81ecfa admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:45.427794 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32095899-6f35-40bd-912c-3a431d81ecfa admin admin] PATH_INFO: `/v3/users/89aa67ee7dff4ad5beb433c399dc1b7f` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:45.429895 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32095899-6f35-40bd-912c-3a431d81ecfa admin admin] RBAC: Authorizing `identity:delete_user(user_id=89aa67ee7dff4ad5beb433c399dc1b7f)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:45.433968 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32095899-6f35-40bd-912c-3a431d81ecfa admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:45.509312 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-32095899-6f35-40bd-912c-3a431d81ecfa admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '89aa67ee7dff4ad5beb433c399dc1b7f'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:45.509982 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-32095899-6f35-40bd-912c-3a431d81ecfa admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '89aa67ee7dff4ad5beb433c399dc1b7f'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:45.513218 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:INSERT=2,keystone:SELECT=91,keystone:DELETE=7 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:56:45.520359 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-32095899-6f35-40bd-912c-3a431d81ecfa admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '89aa67ee7dff4ad5beb433c399dc1b7f'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:45.529071 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-32095899-6f35-40bd-912c-3a431d81ecfa admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '89aa67ee7dff4ad5beb433c399dc1b7f'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:45.529746 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-32095899-6f35-40bd-912c-3a431d81ecfa admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '89aa67ee7dff4ad5beb433c399dc1b7f'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:45.553345 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1078/2143] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 00:56:45 2022] DELETE /identity/v3/users/89aa67ee7dff4ad5beb433c399dc1b7f => generated 0 bytes in 244 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:56:45.558309 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-433a0613-8a10-4f88-8672-3100f2b67f90 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:45.558830 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-433a0613-8a10-4f88-8672-3100f2b67f90 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:45.558830 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-433a0613-8a10-4f88-8672-3100f2b67f90 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:45.584603 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-433a0613-8a10-4f88-8672-3100f2b67f90 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:56:45.585864 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-433a0613-8a10-4f88-8672-3100f2b67f90 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:56:45.636595 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:56:45.672173 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1066/2144] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:56:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:56:45.679068 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32095899-6f35-40bd-912c-3a431d81ecfa admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:45.680604 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-32095899-6f35-40bd-912c-3a431d81ecfa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:56:45.786859 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32095899-6f35-40bd-912c-3a431d81ecfa admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:45.796091 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a96fdbc-ee6b-4e5d-8663-42993e147e04 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:45.801733 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a96fdbc-ee6b-4e5d-8663-42993e147e04 service neutron] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:45.802167 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a96fdbc-ee6b-4e5d-8663-42993e147e04 service neutron] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:45.802450 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a96fdbc-ee6b-4e5d-8663-42993e147e04 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:45.814098 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a96fdbc-ee6b-4e5d-8663-42993e147e04 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:45.815702 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a96fdbc-ee6b-4e5d-8663-42993e147e04 service neutron] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:45.829643 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1079/2145] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:56:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 151 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:56:46.245936 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-433a0613-8a10-4f88-8672-3100f2b67f90 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:46.248879 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-433a0613-8a10-4f88-8672-3100f2b67f90 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:56:46.277606 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-433a0613-8a10-4f88-8672-3100f2b67f90 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:46.295705 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76ab1383-2537-468b-b223-44c2d4dc12c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:46.296537 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76ab1383-2537-468b-b223-44c2d4dc12c3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:46.296898 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76ab1383-2537-468b-b223-44c2d4dc12c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:46.297295 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76ab1383-2537-468b-b223-44c2d4dc12c3 admin admin] PATH_INFO: `/v3/projects/66dbaa2c60594d73ac6fb66012c6061b` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:46.298809 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76ab1383-2537-468b-b223-44c2d4dc12c3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=66dbaa2c60594d73ac6fb66012c6061b)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:46.300933 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76ab1383-2537-468b-b223-44c2d4dc12c3 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:46.367627 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-76ab1383-2537-468b-b223-44c2d4dc12c3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '66dbaa2c60594d73ac6fb66012c6061b', 'request_id': 'req-76ab1383-2537-468b-b223-44c2d4dc12c3'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:46.373907 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-76ab1383-2537-468b-b223-44c2d4dc12c3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '66dbaa2c60594d73ac6fb66012c6061b', 'request_id': 'req-76ab1383-2537-468b-b223-44c2d4dc12c3'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:46.380696 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-76ab1383-2537-468b-b223-44c2d4dc12c3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '66dbaa2c60594d73ac6fb66012c6061b', 'request_id': 'req-76ab1383-2537-468b-b223-44c2d4dc12c3'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:46.395569 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-76ab1383-2537-468b-b223-44c2d4dc12c3 admin admin] The token cache is being invalidate because project 66dbaa2c60594d73ac6fb66012c6061b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81507) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 00:56:46.395808 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-76ab1383-2537-468b-b223-44c2d4dc12c3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:46.396589 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-76ab1383-2537-468b-b223-44c2d4dc12c3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:46.398159 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1067/2146] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 00:56:46 2022] DELETE /identity/v3/projects/66dbaa2c60594d73ac6fb66012c6061b => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:56:46.403142 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a95dee-2078-45ff-9904-bddf509e3dee None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:46.403733 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a95dee-2078-45ff-9904-bddf509e3dee None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:46.404076 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a95dee-2078-45ff-9904-bddf509e3dee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:46.423759 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-53a95dee-2078-45ff-9904-bddf509e3dee None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:56:46.424955 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-53a95dee-2078-45ff-9904-bddf509e3dee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:56:46.494323 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1080/2147] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:56:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:56:46.506816 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4d5b162-4dba-4cfd-81ec-98d8027b6b71 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:46.507370 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4d5b162-4dba-4cfd-81ec-98d8027b6b71 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:46.508084 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4d5b162-4dba-4cfd-81ec-98d8027b6b71 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:46.530395 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-a4d5b162-4dba-4cfd-81ec-98d8027b6b71 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:56:46.531931 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-a4d5b162-4dba-4cfd-81ec-98d8027b6b71 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:56:46.541033 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1068/2148] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:56:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:56:46.546457 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53a95dee-2078-45ff-9904-bddf509e3dee None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:46.557402 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53a95dee-2078-45ff-9904-bddf509e3dee None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:46.562291 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afa1e654-29f1-4b9e-9bff-c3e97014e067 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:46.562980 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afa1e654-29f1-4b9e-9bff-c3e97014e067 admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:46.563256 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afa1e654-29f1-4b9e-9bff-c3e97014e067 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:46.563521 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afa1e654-29f1-4b9e-9bff-c3e97014e067 admin admin] PATH_INFO: `/v3/domains` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:46.564169 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afa1e654-29f1-4b9e-9bff-c3e97014e067 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:56:46.564463 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afa1e654-29f1-4b9e-9bff-c3e97014e067 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:46.566120 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afa1e654-29f1-4b9e-9bff-c3e97014e067 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:46.578706 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1081/2149] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Aug 28 00:56:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:56:46.583002 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4d5b162-4dba-4cfd-81ec-98d8027b6b71 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:46.592626 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4d5b162-4dba-4cfd-81ec-98d8027b6b71 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:46.597652 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de890a87-d7b3-4bc8-886d-928048af2573 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:46.598331 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de890a87-d7b3-4bc8-886d-928048af2573 admin admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:46.598530 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de890a87-d7b3-4bc8-886d-928048af2573 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:46.598811 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de890a87-d7b3-4bc8-886d-928048af2573 admin admin] PATH_INFO: `/v3/projects` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:46.599440 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de890a87-d7b3-4bc8-886d-928048af2573 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:46.601399 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de890a87-d7b3-4bc8-886d-928048af2573 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:46.631390 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1069/2150] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 00:56:46 2022] POST /identity/v3/projects => generated 385 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:56:46.636389 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afa1e654-29f1-4b9e-9bff-c3e97014e067 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:46.646647 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afa1e654-29f1-4b9e-9bff-c3e97014e067 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:46.704042 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:DELETE=3,keystone:SELECT=97 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:56:46.715007 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b451461a-fcd0-4c3d-adbe-ee79faadcd07 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:46.716035 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b451461a-fcd0-4c3d-adbe-ee79faadcd07 admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:46.716312 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b451461a-fcd0-4c3d-adbe-ee79faadcd07 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:46.716580 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b451461a-fcd0-4c3d-adbe-ee79faadcd07 admin admin] PATH_INFO: `/v3/users` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:46.717405 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b451461a-fcd0-4c3d-adbe-ee79faadcd07 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:46.720061 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b451461a-fcd0-4c3d-adbe-ee79faadcd07 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:46.750649 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1082/2151] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 00:56:46 2022] POST /identity/v3/users => generated 293 bytes in 112 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:56:46.751947 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de890a87-d7b3-4bc8-886d-928048af2573 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:46.763132 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de890a87-d7b3-4bc8-886d-928048af2573 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:46.770171 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29a06386-2d0e-4a69-8802-f43eaf76b064 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:46.771143 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29a06386-2d0e-4a69-8802-f43eaf76b064 admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:46.771356 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29a06386-2d0e-4a69-8802-f43eaf76b064 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:46.771774 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29a06386-2d0e-4a69-8802-f43eaf76b064 admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:46.772626 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29a06386-2d0e-4a69-8802-f43eaf76b064 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:46.774535 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29a06386-2d0e-4a69-8802-f43eaf76b064 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:46.787156 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1070/2152] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:56:46 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 28 00:56:46.792433 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b451461a-fcd0-4c3d-adbe-ee79faadcd07 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:46.806551 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b451461a-fcd0-4c3d-adbe-ee79faadcd07 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:46.812549 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10647df9-f9ee-429c-be20-3135f1266ac8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:46.813167 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10647df9-f9ee-429c-be20-3135f1266ac8 admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:46.813486 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10647df9-f9ee-429c-be20-3135f1266ac8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:46.813791 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10647df9-f9ee-429c-be20-3135f1266ac8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:46.814517 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10647df9-f9ee-429c-be20-3135f1266ac8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:46.816406 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10647df9-f9ee-429c-be20-3135f1266ac8 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:46.832073 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1083/2153] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:56:46 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 00:56:46.836854 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29a06386-2d0e-4a69-8802-f43eaf76b064 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:46.849984 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29a06386-2d0e-4a69-8802-f43eaf76b064 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:46.855041 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de8c07a9-704f-459d-8203-1f01033fb8d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:46.855816 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de8c07a9-704f-459d-8203-1f01033fb8d2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:46.856097 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de8c07a9-704f-459d-8203-1f01033fb8d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:46.856370 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de8c07a9-704f-459d-8203-1f01033fb8d2 admin admin] PATH_INFO: `/v3/projects/0981c2250e4b4d0f91c4a46b795d2d1b/users/040bcecafe7a4644af0e660a3416f351/roles/0487ef7e943e4701915cb8ce0f2899d7` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:46.878858 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de8c07a9-704f-459d-8203-1f01033fb8d2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0981c2250e4b4d0f91c4a46b795d2d1b, user_id=040bcecafe7a4644af0e660a3416f351, role_id=0487ef7e943e4701915cb8ce0f2899d7)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:46.881611 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de8c07a9-704f-459d-8203-1f01033fb8d2 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:46.896498 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1071/2154] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 00:56:46 2022] PUT /identity/v3/projects/0981c2250e4b4d0f91c4a46b795d2d1b/users/040bcecafe7a4644af0e660a3416f351/roles/0487ef7e943e4701915cb8ce0f2899d7 => generated 0 bytes in 60 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 28 00:56:46.908564 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b15d439-0c42-4733-9866-83cd863d5a6f None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:46.909209 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b15d439-0c42-4733-9866-83cd863d5a6f None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:46.910068 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b15d439-0c42-4733-9866-83cd863d5a6f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:46.940369 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-2b15d439-0c42-4733-9866-83cd863d5a6f None None] MFA Rules not processed for user `040bcecafe7a4644af0e660a3416f351`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:56:46.941761 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-2b15d439-0c42-4733-9866-83cd863d5a6f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:56:47.023450 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1084/2155] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:56:46 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:56:47.035214 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de8c07a9-704f-459d-8203-1f01033fb8d2 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:47.037315 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-de8c07a9-704f-459d-8203-1f01033fb8d2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:56:47.100521 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de8c07a9-704f-459d-8203-1f01033fb8d2 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:47.104470 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86c94f9d-36e2-434a-bf35-3b0dd6d3301b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:47.105126 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86c94f9d-36e2-434a-bf35-3b0dd6d3301b service cinder] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:47.105409 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86c94f9d-36e2-434a-bf35-3b0dd6d3301b service cinder] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:47.105681 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86c94f9d-36e2-434a-bf35-3b0dd6d3301b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:47.115642 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86c94f9d-36e2-434a-bf35-3b0dd6d3301b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:47.117344 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86c94f9d-36e2-434a-bf35-3b0dd6d3301b service cinder] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:47.130751 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1072/2156] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:56:47 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:56:56.280444 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b15d439-0c42-4733-9866-83cd863d5a6f None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:56.282426 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-2b15d439-0c42-4733-9866-83cd863d5a6f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:56:56.355402 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b15d439-0c42-4733-9866-83cd863d5a6f None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:56.360319 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-527320e4-14c8-4f30-9eba-e5c3d8a66573 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:56.361014 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-527320e4-14c8-4f30-9eba-e5c3d8a66573 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:56.361295 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-527320e4-14c8-4f30-9eba-e5c3d8a66573 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:56.361562 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-527320e4-14c8-4f30-9eba-e5c3d8a66573 admin admin] PATH_INFO: `/v3/users/dec85926b25142529c9ba1fb686dba03` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:56.362875 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-527320e4-14c8-4f30-9eba-e5c3d8a66573 admin admin] RBAC: Authorizing `identity:delete_user(user_id=dec85926b25142529c9ba1fb686dba03)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:56.364833 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-527320e4-14c8-4f30-9eba-e5c3d8a66573 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:56.417641 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-527320e4-14c8-4f30-9eba-e5c3d8a66573 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'dec85926b25142529c9ba1fb686dba03'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:56.418291 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-527320e4-14c8-4f30-9eba-e5c3d8a66573 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'dec85926b25142529c9ba1fb686dba03'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:56.426910 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-527320e4-14c8-4f30-9eba-e5c3d8a66573 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'dec85926b25142529c9ba1fb686dba03'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:56.437950 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-527320e4-14c8-4f30-9eba-e5c3d8a66573 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'dec85926b25142529c9ba1fb686dba03'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:56.438549 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-527320e4-14c8-4f30-9eba-e5c3d8a66573 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'dec85926b25142529c9ba1fb686dba03'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:56.443656 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=88,keystone:INSERT=4,keystone:DELETE=8 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:56:56.459456 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1085/2157] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 00:56:56 2022] DELETE /identity/v3/users/dec85926b25142529c9ba1fb686dba03 => generated 0 bytes in 180 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:56:56.468394 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bca8619-7afd-428a-86bd-dca2fe28246e None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:56.468394 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bca8619-7afd-428a-86bd-dca2fe28246e None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:56.468394 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bca8619-7afd-428a-86bd-dca2fe28246e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:56.473299 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:56:56.486445 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-9bca8619-7afd-428a-86bd-dca2fe28246e None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:56:56.488384 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-9bca8619-7afd-428a-86bd-dca2fe28246e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:56:56.561455 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1073/2158] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:56:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:56:56.571909 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-527320e4-14c8-4f30-9eba-e5c3d8a66573 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:56.573815 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-527320e4-14c8-4f30-9eba-e5c3d8a66573 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:56:56.636458 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-527320e4-14c8-4f30-9eba-e5c3d8a66573 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:56.641056 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-037e8e90-5424-44fa-bf5b-18661e160c63 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:56.641957 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-037e8e90-5424-44fa-bf5b-18661e160c63 service neutron] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:56.641957 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-037e8e90-5424-44fa-bf5b-18661e160c63 service neutron] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:56.642223 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-037e8e90-5424-44fa-bf5b-18661e160c63 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:56.652994 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-037e8e90-5424-44fa-bf5b-18661e160c63 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:56.654571 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-037e8e90-5424-44fa-bf5b-18661e160c63 service neutron] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:56.664936 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1086/2159] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:56:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:56:56.959901 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bca8619-7afd-428a-86bd-dca2fe28246e None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:56.961596 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-9bca8619-7afd-428a-86bd-dca2fe28246e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:56:56.978315 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bca8619-7afd-428a-86bd-dca2fe28246e None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:56.984922 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5e5b037-04c6-43a0-8dae-cbd3c72b271f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:56.985671 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5e5b037-04c6-43a0-8dae-cbd3c72b271f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:56.985887 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5e5b037-04c6-43a0-8dae-cbd3c72b271f admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:56.986153 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5e5b037-04c6-43a0-8dae-cbd3c72b271f admin admin] PATH_INFO: `/v3/projects/da2beaaf1c7846a5aac4a75fd04461ce` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:56.987597 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5e5b037-04c6-43a0-8dae-cbd3c72b271f admin admin] RBAC: Authorizing `identity:delete_project(project_id=da2beaaf1c7846a5aac4a75fd04461ce)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:56.989698 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5e5b037-04c6-43a0-8dae-cbd3c72b271f admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:57.042854 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-c5e5b037-04c6-43a0-8dae-cbd3c72b271f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'da2beaaf1c7846a5aac4a75fd04461ce', 'request_id': 'req-c5e5b037-04c6-43a0-8dae-cbd3c72b271f'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:57.049507 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-c5e5b037-04c6-43a0-8dae-cbd3c72b271f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'da2beaaf1c7846a5aac4a75fd04461ce', 'request_id': 'req-c5e5b037-04c6-43a0-8dae-cbd3c72b271f'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:57.055950 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-c5e5b037-04c6-43a0-8dae-cbd3c72b271f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'da2beaaf1c7846a5aac4a75fd04461ce', 'request_id': 'req-c5e5b037-04c6-43a0-8dae-cbd3c72b271f'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:57.072031 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-c5e5b037-04c6-43a0-8dae-cbd3c72b271f admin admin] The token cache is being invalidate because project da2beaaf1c7846a5aac4a75fd04461ce was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81507) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 00:56:57.072281 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-c5e5b037-04c6-43a0-8dae-cbd3c72b271f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:57.073455 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-c5e5b037-04c6-43a0-8dae-cbd3c72b271f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:57.075068 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1074/2160] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 00:56:56 2022] DELETE /identity/v3/projects/da2beaaf1c7846a5aac4a75fd04461ce => generated 0 bytes in 116 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:56:57.875704 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-037e8e90-5424-44fa-bf5b-18661e160c63 service neutron] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:57.877649 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-037e8e90-5424-44fa-bf5b-18661e160c63 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:56:57.892622 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-037e8e90-5424-44fa-bf5b-18661e160c63 service neutron] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:57.955896 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5afd64c0-6cfe-4085-91d2-9a334355e45f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:57.956517 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5afd64c0-6cfe-4085-91d2-9a334355e45f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:57.956792 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5afd64c0-6cfe-4085-91d2-9a334355e45f admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:57.957090 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5afd64c0-6cfe-4085-91d2-9a334355e45f admin admin] PATH_INFO: `/v3/users/040bcecafe7a4644af0e660a3416f351` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:57.959088 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5afd64c0-6cfe-4085-91d2-9a334355e45f admin admin] RBAC: Authorizing `identity:delete_user(user_id=040bcecafe7a4644af0e660a3416f351)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:57.963635 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5afd64c0-6cfe-4085-91d2-9a334355e45f admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:58.022108 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-5afd64c0-6cfe-4085-91d2-9a334355e45f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '040bcecafe7a4644af0e660a3416f351'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:58.022784 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-5afd64c0-6cfe-4085-91d2-9a334355e45f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '040bcecafe7a4644af0e660a3416f351'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:58.032701 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-5afd64c0-6cfe-4085-91d2-9a334355e45f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '040bcecafe7a4644af0e660a3416f351'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:58.049356 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-5afd64c0-6cfe-4085-91d2-9a334355e45f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '040bcecafe7a4644af0e660a3416f351'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:58.050111 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-5afd64c0-6cfe-4085-91d2-9a334355e45f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '040bcecafe7a4644af0e660a3416f351'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:58.071125 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1087/2161] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 00:56:57 2022] DELETE /identity/v3/users/040bcecafe7a4644af0e660a3416f351 => generated 0 bytes in 196 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:56:58.076214 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad301ba2-709e-4a0c-9e44-14615a9a8146 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:58.076712 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad301ba2-709e-4a0c-9e44-14615a9a8146 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:58.076807 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad301ba2-709e-4a0c-9e44-14615a9a8146 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:58.097138 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-ad301ba2-709e-4a0c-9e44-14615a9a8146 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:56:58.098382 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-ad301ba2-709e-4a0c-9e44-14615a9a8146 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:56:58.168262 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1075/2162] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:56:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 28 00:56:58.179552 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5afd64c0-6cfe-4085-91d2-9a334355e45f admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:58.181494 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-5afd64c0-6cfe-4085-91d2-9a334355e45f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:56:58.193198 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:DELETE=12,keystone:SELECT=87,keystone:INSERT=1 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:56:58.252232 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5afd64c0-6cfe-4085-91d2-9a334355e45f admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:58.256601 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e77341d1-42f4-4c15-977c-2ffad048aade service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:58.257500 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e77341d1-42f4-4c15-977c-2ffad048aade service neutron] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:58.257784 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e77341d1-42f4-4c15-977c-2ffad048aade service neutron] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:58.258053 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e77341d1-42f4-4c15-977c-2ffad048aade service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:58.273317 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e77341d1-42f4-4c15-977c-2ffad048aade service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:58.274767 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e77341d1-42f4-4c15-977c-2ffad048aade service neutron] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:58.285623 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1088/2163] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:56:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:56:58.579336 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad301ba2-709e-4a0c-9e44-14615a9a8146 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:58.580977 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-ad301ba2-709e-4a0c-9e44-14615a9a8146 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:56:58.593995 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad301ba2-709e-4a0c-9e44-14615a9a8146 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:58.600157 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f9bdb22-fc31-46f4-991d-811d3d5c4381 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:58.600867 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f9bdb22-fc31-46f4-991d-811d3d5c4381 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:58.601186 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f9bdb22-fc31-46f4-991d-811d3d5c4381 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:58.601457 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f9bdb22-fc31-46f4-991d-811d3d5c4381 admin admin] PATH_INFO: `/v3/projects/0981c2250e4b4d0f91c4a46b795d2d1b` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:58.602725 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f9bdb22-fc31-46f4-991d-811d3d5c4381 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0981c2250e4b4d0f91c4a46b795d2d1b)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:58.604669 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f9bdb22-fc31-46f4-991d-811d3d5c4381 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:58.633558 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:56:58.662118 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-0f9bdb22-fc31-46f4-991d-811d3d5c4381 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0981c2250e4b4d0f91c4a46b795d2d1b', 'request_id': 'req-0f9bdb22-fc31-46f4-991d-811d3d5c4381'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:58.668269 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-0f9bdb22-fc31-46f4-991d-811d3d5c4381 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0981c2250e4b4d0f91c4a46b795d2d1b', 'request_id': 'req-0f9bdb22-fc31-46f4-991d-811d3d5c4381'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:58.675132 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-0f9bdb22-fc31-46f4-991d-811d3d5c4381 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0981c2250e4b4d0f91c4a46b795d2d1b', 'request_id': 'req-0f9bdb22-fc31-46f4-991d-811d3d5c4381'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:58.688686 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-0f9bdb22-fc31-46f4-991d-811d3d5c4381 admin admin] The token cache is being invalidate because project 0981c2250e4b4d0f91c4a46b795d2d1b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81507) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 00:56:58.688991 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-0f9bdb22-fc31-46f4-991d-811d3d5c4381 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:58.689688 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-0f9bdb22-fc31-46f4-991d-811d3d5c4381 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:56:58.691252 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1076/2164] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 00:56:58 2022] DELETE /identity/v3/projects/0981c2250e4b4d0f91c4a46b795d2d1b => generated 0 bytes in 112 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:56:58.701891 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-217cc320-cda4-4f09-aa30-e5faea8bc8b6 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:58.702319 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-217cc320-cda4-4f09-aa30-e5faea8bc8b6 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:58.702614 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-217cc320-cda4-4f09-aa30-e5faea8bc8b6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:58.724585 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-217cc320-cda4-4f09-aa30-e5faea8bc8b6 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:56:58.725930 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-217cc320-cda4-4f09-aa30-e5faea8bc8b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:56:58.787573 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1089/2165] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:56:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:56:58.796037 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afc40146-32ca-4a87-b101-685117d89a5d None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:58.796320 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afc40146-32ca-4a87-b101-685117d89a5d None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:58.796873 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afc40146-32ca-4a87-b101-685117d89a5d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:58.817443 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-afc40146-32ca-4a87-b101-685117d89a5d None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:56:58.818675 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-afc40146-32ca-4a87-b101-685117d89a5d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:56:58.827431 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1077/2166] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:56:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:56:58.832283 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-217cc320-cda4-4f09-aa30-e5faea8bc8b6 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:58.842757 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-217cc320-cda4-4f09-aa30-e5faea8bc8b6 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:58.848034 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4fd5c47-c5a0-4ec3-9e34-61b878870eca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:58.848658 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4fd5c47-c5a0-4ec3-9e34-61b878870eca admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:58.848936 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4fd5c47-c5a0-4ec3-9e34-61b878870eca admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:58.849233 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4fd5c47-c5a0-4ec3-9e34-61b878870eca admin admin] PATH_INFO: `/v3/domains` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:58.849886 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4fd5c47-c5a0-4ec3-9e34-61b878870eca admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:56:58.850182 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4fd5c47-c5a0-4ec3-9e34-61b878870eca admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:58.851793 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4fd5c47-c5a0-4ec3-9e34-61b878870eca admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:58.864499 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1090/2167] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Aug 28 00:56:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:56:58.869219 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afc40146-32ca-4a87-b101-685117d89a5d None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:58.879186 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afc40146-32ca-4a87-b101-685117d89a5d None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:58.884168 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5adcccaa-c885-4b78-899b-3b2786548da3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:58.884784 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5adcccaa-c885-4b78-899b-3b2786548da3 admin admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:58.885086 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5adcccaa-c885-4b78-899b-3b2786548da3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:58.885357 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5adcccaa-c885-4b78-899b-3b2786548da3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:58.885998 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5adcccaa-c885-4b78-899b-3b2786548da3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:58.888114 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5adcccaa-c885-4b78-899b-3b2786548da3 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:58.920058 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1078/2168] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 00:56:58 2022] POST /identity/v3/projects => generated 373 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:56:58.924558 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4fd5c47-c5a0-4ec3-9e34-61b878870eca admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:58.934795 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4fd5c47-c5a0-4ec3-9e34-61b878870eca admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:59.020253 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad158294-8477-4347-b64b-c8800a3dd14b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:59.020860 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad158294-8477-4347-b64b-c8800a3dd14b admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:59.021191 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad158294-8477-4347-b64b-c8800a3dd14b admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:59.021459 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad158294-8477-4347-b64b-c8800a3dd14b admin admin] PATH_INFO: `/v3/users` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:59.022104 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad158294-8477-4347-b64b-c8800a3dd14b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:59.024022 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad158294-8477-4347-b64b-c8800a3dd14b admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:59.049083 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1091/2169] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 00:56:58 2022] POST /identity/v3/users => generated 287 bytes in 122 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 28 00:56:59.050301 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5adcccaa-c885-4b78-899b-3b2786548da3 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:59.061237 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5adcccaa-c885-4b78-899b-3b2786548da3 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:59.066322 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c6ea11c-f6bf-4e83-a712-5236d346f886 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:59.066849 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c6ea11c-f6bf-4e83-a712-5236d346f886 admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:59.067144 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c6ea11c-f6bf-4e83-a712-5236d346f886 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:59.067401 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c6ea11c-f6bf-4e83-a712-5236d346f886 admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:59.068040 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c6ea11c-f6bf-4e83-a712-5236d346f886 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:59.069676 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c6ea11c-f6bf-4e83-a712-5236d346f886 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:59.081041 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1079/2170] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:56:59 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 00:56:59.085395 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad158294-8477-4347-b64b-c8800a3dd14b admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:59.098388 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad158294-8477-4347-b64b-c8800a3dd14b admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:59.104453 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1da67d85-734f-4ce6-8773-ac06a2a6933c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:59.105079 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1da67d85-734f-4ce6-8773-ac06a2a6933c admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:59.105356 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1da67d85-734f-4ce6-8773-ac06a2a6933c admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:59.105630 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1da67d85-734f-4ce6-8773-ac06a2a6933c admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:59.106266 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1da67d85-734f-4ce6-8773-ac06a2a6933c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:59.107862 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1da67d85-734f-4ce6-8773-ac06a2a6933c admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:59.120428 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1092/2171] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:56:59 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 00:56:59.131694 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c6ea11c-f6bf-4e83-a712-5236d346f886 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:59.142016 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c6ea11c-f6bf-4e83-a712-5236d346f886 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:59.147169 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-918f5e46-6d1d-4ef4-b35b-994860f875b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:59.147867 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-918f5e46-6d1d-4ef4-b35b-994860f875b0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:59.148152 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-918f5e46-6d1d-4ef4-b35b-994860f875b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:59.148424 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-918f5e46-6d1d-4ef4-b35b-994860f875b0 admin admin] PATH_INFO: `/v3/projects/315d6d72ee5844edb203b740ab2d08ce/users/59b04859082547149bf2776aefa63b43/roles/0487ef7e943e4701915cb8ce0f2899d7` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:59.166293 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-918f5e46-6d1d-4ef4-b35b-994860f875b0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=315d6d72ee5844edb203b740ab2d08ce, user_id=59b04859082547149bf2776aefa63b43, role_id=0487ef7e943e4701915cb8ce0f2899d7)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:59.169045 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-918f5e46-6d1d-4ef4-b35b-994860f875b0 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:59.185956 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1080/2172] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 00:56:59 2022] PUT /identity/v3/projects/315d6d72ee5844edb203b740ab2d08ce/users/59b04859082547149bf2776aefa63b43/roles/0487ef7e943e4701915cb8ce0f2899d7 => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:56:59.191488 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1141906-0a83-4c5d-95d4-9777cc820693 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:59.191763 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1141906-0a83-4c5d-95d4-9777cc820693 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:59.192041 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1141906-0a83-4c5d-95d4-9777cc820693 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:59.212051 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-c1141906-0a83-4c5d-95d4-9777cc820693 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:56:59.213302 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-c1141906-0a83-4c5d-95d4-9777cc820693 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:56:59.225585 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:56:59.282588 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1093/2173] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:56:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:56:59.295508 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-918f5e46-6d1d-4ef4-b35b-994860f875b0 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:56:59.297411 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-918f5e46-6d1d-4ef4-b35b-994860f875b0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:56:59.362046 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-918f5e46-6d1d-4ef4-b35b-994860f875b0 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:56:59.365955 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61e4bb18-2fc8-41e5-b420-9c601b667ff3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:56:59.366564 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61e4bb18-2fc8-41e5-b420-9c601b667ff3 service neutron] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:56:59.367137 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61e4bb18-2fc8-41e5-b420-9c601b667ff3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:56:59.367137 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61e4bb18-2fc8-41e5-b420-9c601b667ff3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:56:59.376678 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61e4bb18-2fc8-41e5-b420-9c601b667ff3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:56:59.378610 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61e4bb18-2fc8-41e5-b420-9c601b667ff3 service neutron] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:56:59.388629 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1081/2174] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:56:59 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 28 00:57:00.218724 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a94f75b-b8bf-439d-9ff0-8c4e802f6878 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:57:00.219293 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a94f75b-b8bf-439d-9ff0-8c4e802f6878 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:57:00.219293 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a94f75b-b8bf-439d-9ff0-8c4e802f6878 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:57:00.250264 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-8a94f75b-b8bf-439d-9ff0-8c4e802f6878 None None] MFA Rules not processed for user `59b04859082547149bf2776aefa63b43`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:57:00.251523 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-8a94f75b-b8bf-439d-9ff0-8c4e802f6878 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:57:00.320494 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1094/2175] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:57:00 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:57:00.331952 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61e4bb18-2fc8-41e5-b420-9c601b667ff3 service neutron] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:57:00.333543 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-61e4bb18-2fc8-41e5-b420-9c601b667ff3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:57:00.397670 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61e4bb18-2fc8-41e5-b420-9c601b667ff3 service neutron] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:57:00.401453 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16829ba9-a103-47d4-9b84-8aa37a74aea5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:57:00.402069 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16829ba9-a103-47d4-9b84-8aa37a74aea5 service cinder] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:57:00.402344 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16829ba9-a103-47d4-9b84-8aa37a74aea5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:57:00.402611 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16829ba9-a103-47d4-9b84-8aa37a74aea5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:57:00.408225 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=91,keystone:DELETE=6,keystone:INSERT=3 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:57:00.413717 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16829ba9-a103-47d4-9b84-8aa37a74aea5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:57:00.415319 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16829ba9-a103-47d4-9b84-8aa37a74aea5 service cinder] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:57:00.426683 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1082/2176] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:57:00 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:57:02.889306 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8524d3f3-1082-4cfe-9b6e-e869e81ad859 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:57:02.889856 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8524d3f3-1082-4cfe-9b6e-e869e81ad859 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:57:02.889856 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8524d3f3-1082-4cfe-9b6e-e869e81ad859 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:57:02.910540 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-8524d3f3-1082-4cfe-9b6e-e869e81ad859 None None] MFA Rules not processed for user `59b04859082547149bf2776aefa63b43`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:57:02.911748 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-8524d3f3-1082-4cfe-9b6e-e869e81ad859 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:57:02.919823 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1095/2177] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:57:02 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:57:04.084571 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca818042-0eb1-4025-9084-65cf36c81ebe None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:57:04.085187 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca818042-0eb1-4025-9084-65cf36c81ebe None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:57:04.085187 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca818042-0eb1-4025-9084-65cf36c81ebe None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:57:04.086739 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1083/2178] 10.222.0.42 () {54 vars in 921 bytes} [Sun Aug 28 00:57:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:57:04.124274 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dd19a61-9227-46a2-b748-283f7eb4d6f4 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:57:04.125211 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dd19a61-9227-46a2-b748-283f7eb4d6f4 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:57:04.125498 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dd19a61-9227-46a2-b748-283f7eb4d6f4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:57:04.149362 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-9dd19a61-9227-46a2-b748-283f7eb4d6f4 None None] MFA Rules not processed for user `c2a3834fa6894dada3291e4781bbd553`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:57:04.150985 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-9dd19a61-9227-46a2-b748-283f7eb4d6f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:57:04.242200 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1096/2179] 10.222.0.42 () {58 vars in 1018 bytes} [Sun Aug 28 00:57:04 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:57:04.269075 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca818042-0eb1-4025-9084-65cf36c81ebe None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:57:04.271331 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-ca818042-0eb1-4025-9084-65cf36c81ebe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:57:04.289598 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca818042-0eb1-4025-9084-65cf36c81ebe None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:57:04.295306 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fe89237-69d1-4b46-9408-e321c14e0fbe service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2a3834fa6894dada3291e4781bbd553', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:57:04.295931 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fe89237-69d1-4b46-9408-e321c14e0fbe service placement] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:57:04.296352 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fe89237-69d1-4b46-9408-e321c14e0fbe service placement] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:57:04.296628 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fe89237-69d1-4b46-9408-e321c14e0fbe service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:57:04.306124 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fe89237-69d1-4b46-9408-e321c14e0fbe service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:57:04.307719 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fe89237-69d1-4b46-9408-e321c14e0fbe service placement] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:57:04.318130 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1084/2180] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:57:04 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 50 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 28 00:57:04.864590 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dd19a61-9227-46a2-b748-283f7eb4d6f4 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:57:04.866387 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-9dd19a61-9227-46a2-b748-283f7eb4d6f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:57:04.880079 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dd19a61-9227-46a2-b748-283f7eb4d6f4 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:57:04.886426 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-126cc852-a59c-4161-b9ac-3a935176ab29 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:57:04.886426 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-126cc852-a59c-4161-b9ac-3a935176ab29 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:57:04.886671 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-126cc852-a59c-4161-b9ac-3a935176ab29 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:57:04.887121 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-126cc852-a59c-4161-b9ac-3a935176ab29 admin admin] PATH_INFO: `/v3/users/69aa1d53c9fa4a9a942d18c15886c93f` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:57:04.888508 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-126cc852-a59c-4161-b9ac-3a935176ab29 admin admin] RBAC: Authorizing `identity:delete_user(user_id=69aa1d53c9fa4a9a942d18c15886c93f)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:57:04.890715 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-126cc852-a59c-4161-b9ac-3a935176ab29 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:57:04.933023 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:57:04.951966 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-126cc852-a59c-4161-b9ac-3a935176ab29 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '69aa1d53c9fa4a9a942d18c15886c93f'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:57:04.953059 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-126cc852-a59c-4161-b9ac-3a935176ab29 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '69aa1d53c9fa4a9a942d18c15886c93f'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:57:04.963352 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-126cc852-a59c-4161-b9ac-3a935176ab29 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '69aa1d53c9fa4a9a942d18c15886c93f'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:57:04.973814 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-126cc852-a59c-4161-b9ac-3a935176ab29 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '69aa1d53c9fa4a9a942d18c15886c93f'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:57:04.973814 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-126cc852-a59c-4161-b9ac-3a935176ab29 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '69aa1d53c9fa4a9a942d18c15886c93f'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:57:04.994639 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1097/2181] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 00:57:04 2022] DELETE /identity/v3/users/69aa1d53c9fa4a9a942d18c15886c93f => generated 0 bytes in 131 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:57:05.469890 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fe89237-69d1-4b46-9408-e321c14e0fbe service placement] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:57:05.481911 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fe89237-69d1-4b46-9408-e321c14e0fbe service placement] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:57:05.545661 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-841b91a7-02e2-4969-90d5-9daaafad8832 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:57:05.546514 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-841b91a7-02e2-4969-90d5-9daaafad8832 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:57:05.546907 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-841b91a7-02e2-4969-90d5-9daaafad8832 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:57:05.547246 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-841b91a7-02e2-4969-90d5-9daaafad8832 admin admin] PATH_INFO: `/v3/projects/bc3d0dc67a6f44c59cb2007ffe156dec` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:57:05.548440 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-841b91a7-02e2-4969-90d5-9daaafad8832 admin admin] RBAC: Authorizing `identity:delete_project(project_id=bc3d0dc67a6f44c59cb2007ffe156dec)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:57:05.550751 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-841b91a7-02e2-4969-90d5-9daaafad8832 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:57:05.603697 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-841b91a7-02e2-4969-90d5-9daaafad8832 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bc3d0dc67a6f44c59cb2007ffe156dec', 'request_id': 'req-841b91a7-02e2-4969-90d5-9daaafad8832'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:57:05.611212 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-841b91a7-02e2-4969-90d5-9daaafad8832 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bc3d0dc67a6f44c59cb2007ffe156dec', 'request_id': 'req-841b91a7-02e2-4969-90d5-9daaafad8832'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:57:05.617878 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-841b91a7-02e2-4969-90d5-9daaafad8832 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bc3d0dc67a6f44c59cb2007ffe156dec', 'request_id': 'req-841b91a7-02e2-4969-90d5-9daaafad8832'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:57:05.632366 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-841b91a7-02e2-4969-90d5-9daaafad8832 admin admin] The token cache is being invalidate because project bc3d0dc67a6f44c59cb2007ffe156dec was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81507) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 00:57:05.633106 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-841b91a7-02e2-4969-90d5-9daaafad8832 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:57:05.633513 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-841b91a7-02e2-4969-90d5-9daaafad8832 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:57:05.635360 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1085/2182] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 00:57:05 2022] DELETE /identity/v3/projects/bc3d0dc67a6f44c59cb2007ffe156dec => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 28 00:57:05.639775 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-126cc852-a59c-4161-b9ac-3a935176ab29 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:57:05.642358 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-126cc852-a59c-4161-b9ac-3a935176ab29 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:57:05.655868 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-126cc852-a59c-4161-b9ac-3a935176ab29 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:57:05.721390 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-311dd75c-3d6c-446f-b928-dbdad789e00a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:57:05.722275 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-311dd75c-3d6c-446f-b928-dbdad789e00a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:57:05.722471 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-311dd75c-3d6c-446f-b928-dbdad789e00a admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:57:05.722852 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-311dd75c-3d6c-446f-b928-dbdad789e00a admin admin] PATH_INFO: `/v3/users/080d9f9f889d457fb08929c3e1db4109` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:57:05.724127 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-311dd75c-3d6c-446f-b928-dbdad789e00a admin admin] RBAC: Authorizing `identity:delete_user(user_id=080d9f9f889d457fb08929c3e1db4109)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:57:05.726149 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-311dd75c-3d6c-446f-b928-dbdad789e00a admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:57:05.791113 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-311dd75c-3d6c-446f-b928-dbdad789e00a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '080d9f9f889d457fb08929c3e1db4109'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:57:05.792229 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-311dd75c-3d6c-446f-b928-dbdad789e00a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '080d9f9f889d457fb08929c3e1db4109'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:57:05.802337 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-311dd75c-3d6c-446f-b928-dbdad789e00a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '080d9f9f889d457fb08929c3e1db4109'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:57:05.810028 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-311dd75c-3d6c-446f-b928-dbdad789e00a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '080d9f9f889d457fb08929c3e1db4109'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:57:05.810762 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-311dd75c-3d6c-446f-b928-dbdad789e00a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '080d9f9f889d457fb08929c3e1db4109'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:57:05.835653 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1098/2183] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 00:57:05 2022] DELETE /identity/v3/users/080d9f9f889d457fb08929c3e1db4109 => generated 0 bytes in 194 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 28 00:57:06.035553 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-841b91a7-02e2-4969-90d5-9daaafad8832 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:57:06.038991 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-841b91a7-02e2-4969-90d5-9daaafad8832 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:57:06.076798 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-841b91a7-02e2-4969-90d5-9daaafad8832 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:57:06.148353 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-018ce551-909f-4ad4-8fed-9b7996476edd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:57:06.149031 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-018ce551-909f-4ad4-8fed-9b7996476edd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:57:06.149314 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-018ce551-909f-4ad4-8fed-9b7996476edd admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:57:06.149579 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-018ce551-909f-4ad4-8fed-9b7996476edd admin admin] PATH_INFO: `/v3/projects/d5263ed3d2ee43f4b99d605d054fa20e` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:57:06.150718 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-018ce551-909f-4ad4-8fed-9b7996476edd admin admin] RBAC: Authorizing `identity:delete_project(project_id=d5263ed3d2ee43f4b99d605d054fa20e)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:57:06.152677 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-018ce551-909f-4ad4-8fed-9b7996476edd admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:57:06.165193 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:57:06.208941 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-018ce551-909f-4ad4-8fed-9b7996476edd admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd5263ed3d2ee43f4b99d605d054fa20e', 'request_id': 'req-018ce551-909f-4ad4-8fed-9b7996476edd'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:57:06.215857 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-018ce551-909f-4ad4-8fed-9b7996476edd admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd5263ed3d2ee43f4b99d605d054fa20e', 'request_id': 'req-018ce551-909f-4ad4-8fed-9b7996476edd'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:57:06.223599 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-018ce551-909f-4ad4-8fed-9b7996476edd admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd5263ed3d2ee43f4b99d605d054fa20e', 'request_id': 'req-018ce551-909f-4ad4-8fed-9b7996476edd'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:57:06.239913 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-018ce551-909f-4ad4-8fed-9b7996476edd admin admin] The token cache is being invalidate because project d5263ed3d2ee43f4b99d605d054fa20e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81507) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 00:57:06.240199 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-018ce551-909f-4ad4-8fed-9b7996476edd admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:57:06.241031 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-018ce551-909f-4ad4-8fed-9b7996476edd admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:57:06.242607 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1086/2184] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 00:57:06 2022] DELETE /identity/v3/projects/d5263ed3d2ee43f4b99d605d054fa20e => generated 0 bytes in 208 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:57:06.248024 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d9194f-494c-4fc2-927a-fd33695410c4 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:57:06.248397 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d9194f-494c-4fc2-927a-fd33695410c4 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:57:06.248674 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d9194f-494c-4fc2-927a-fd33695410c4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:57:06.269836 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-91d9194f-494c-4fc2-927a-fd33695410c4 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:57:06.271079 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-91d9194f-494c-4fc2-927a-fd33695410c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:57:06.293987 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=83,keystone:DELETE=15,keystone:INSERT=2 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:57:06.341330 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1099/2185] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:57:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:57:06.349859 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-824cf8dc-e455-4769-858c-861f2ddb73df None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:57:06.350228 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-824cf8dc-e455-4769-858c-861f2ddb73df None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:57:06.350570 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-824cf8dc-e455-4769-858c-861f2ddb73df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:57:06.372031 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-824cf8dc-e455-4769-858c-861f2ddb73df None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:57:06.373283 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-824cf8dc-e455-4769-858c-861f2ddb73df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:57:06.382647 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1087/2186] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:57:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:57:06.387115 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91d9194f-494c-4fc2-927a-fd33695410c4 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:57:06.397924 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91d9194f-494c-4fc2-927a-fd33695410c4 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:57:06.411851 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-488b14c9-3da6-41d9-9653-abafccda554a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:57:06.411851 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-488b14c9-3da6-41d9-9653-abafccda554a admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:57:06.411851 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-488b14c9-3da6-41d9-9653-abafccda554a admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:57:06.411851 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-488b14c9-3da6-41d9-9653-abafccda554a admin admin] PATH_INFO: `/v3/domains` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:57:06.412225 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-488b14c9-3da6-41d9-9653-abafccda554a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:57:06.412225 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-488b14c9-3da6-41d9-9653-abafccda554a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:57:06.412225 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-488b14c9-3da6-41d9-9653-abafccda554a admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:57:06.428424 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1100/2187] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Aug 28 00:57:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:57:06.429752 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-824cf8dc-e455-4769-858c-861f2ddb73df None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:57:06.461305 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-824cf8dc-e455-4769-858c-861f2ddb73df None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:57:06.467278 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7df3937e-4716-4d78-8b21-774795bd4098 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:57:06.467897 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7df3937e-4716-4d78-8b21-774795bd4098 admin admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:57:06.468205 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7df3937e-4716-4d78-8b21-774795bd4098 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:57:06.468476 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7df3937e-4716-4d78-8b21-774795bd4098 admin admin] PATH_INFO: `/v3/projects` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:57:06.470093 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7df3937e-4716-4d78-8b21-774795bd4098 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:57:06.472408 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7df3937e-4716-4d78-8b21-774795bd4098 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:57:06.503319 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1088/2188] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 00:57:06 2022] POST /identity/v3/projects => generated 373 bytes in 74 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:57:06.507565 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-488b14c9-3da6-41d9-9653-abafccda554a admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:57:06.518021 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-488b14c9-3da6-41d9-9653-abafccda554a admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:57:06.582003 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28056102-3298-406b-ab46-72d7d152daf7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:57:06.582631 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28056102-3298-406b-ab46-72d7d152daf7 admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:57:06.582908 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28056102-3298-406b-ab46-72d7d152daf7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:57:06.583177 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28056102-3298-406b-ab46-72d7d152daf7 admin admin] PATH_INFO: `/v3/users` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:57:06.583873 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28056102-3298-406b-ab46-72d7d152daf7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:57:06.585908 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28056102-3298-406b-ab46-72d7d152daf7 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:57:06.611161 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1101/2189] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 00:57:06 2022] POST /identity/v3/users => generated 287 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 28 00:57:06.615683 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7df3937e-4716-4d78-8b21-774795bd4098 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:57:06.628692 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7df3937e-4716-4d78-8b21-774795bd4098 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:57:06.633440 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f5983c3-0792-48df-88e3-d849c932d889 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:57:06.634017 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f5983c3-0792-48df-88e3-d849c932d889 admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:57:06.634292 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f5983c3-0792-48df-88e3-d849c932d889 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:57:06.634560 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f5983c3-0792-48df-88e3-d849c932d889 admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:57:06.635197 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f5983c3-0792-48df-88e3-d849c932d889 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:57:06.636833 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f5983c3-0792-48df-88e3-d849c932d889 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:57:06.647665 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1089/2190] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:57:06 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 28 00:57:06.652660 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28056102-3298-406b-ab46-72d7d152daf7 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:57:06.663551 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28056102-3298-406b-ab46-72d7d152daf7 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:57:06.668506 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0f8bbfd-75e4-4f49-8134-170048e530d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:57:06.669146 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0f8bbfd-75e4-4f49-8134-170048e530d7 admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:57:06.669423 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0f8bbfd-75e4-4f49-8134-170048e530d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:57:06.669694 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0f8bbfd-75e4-4f49-8134-170048e530d7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:57:06.670334 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0f8bbfd-75e4-4f49-8134-170048e530d7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:57:06.671977 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0f8bbfd-75e4-4f49-8134-170048e530d7 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:57:06.683637 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1102/2191] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:57:06 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 00:57:06.687994 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f5983c3-0792-48df-88e3-d849c932d889 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:57:06.698790 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f5983c3-0792-48df-88e3-d849c932d889 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:57:06.703525 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5d088d8-15cf-4a9f-95b2-ff274d808c3a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:57:06.704191 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5d088d8-15cf-4a9f-95b2-ff274d808c3a admin admin] REQUEST_METHOD: `PUT` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:57:06.704468 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5d088d8-15cf-4a9f-95b2-ff274d808c3a admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:57:06.704742 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5d088d8-15cf-4a9f-95b2-ff274d808c3a admin admin] PATH_INFO: `/v3/projects/4b4cf8ca6866462fb1eaa31c3fe9341b/users/c9f1cd3fee77460d879c55cab26c4a7e/roles/0487ef7e943e4701915cb8ce0f2899d7` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:57:06.722345 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5d088d8-15cf-4a9f-95b2-ff274d808c3a admin admin] RBAC: Authorizing `identity:create_grant(project_id=4b4cf8ca6866462fb1eaa31c3fe9341b, user_id=c9f1cd3fee77460d879c55cab26c4a7e, role_id=0487ef7e943e4701915cb8ce0f2899d7)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:57:06.732555 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5d088d8-15cf-4a9f-95b2-ff274d808c3a admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:57:06.751831 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1090/2192] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 00:57:06 2022] PUT /identity/v3/projects/4b4cf8ca6866462fb1eaa31c3fe9341b/users/c9f1cd3fee77460d879c55cab26c4a7e/roles/0487ef7e943e4701915cb8ce0f2899d7 => generated 0 bytes in 64 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 28 00:57:06.762548 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f51f565-e65d-4aff-8cdd-51fa997afc53 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:57:06.762830 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f51f565-e65d-4aff-8cdd-51fa997afc53 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:57:06.763100 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f51f565-e65d-4aff-8cdd-51fa997afc53 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:57:06.791942 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-3f51f565-e65d-4aff-8cdd-51fa997afc53 None None] MFA Rules not processed for user `c9f1cd3fee77460d879c55cab26c4a7e`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:57:06.793479 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-3f51f565-e65d-4aff-8cdd-51fa997afc53 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:57:06.862695 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1103/2193] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:57:06 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:57:06.874935 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5d088d8-15cf-4a9f-95b2-ff274d808c3a admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:57:06.876929 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-b5d088d8-15cf-4a9f-95b2-ff274d808c3a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:57:06.936762 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5d088d8-15cf-4a9f-95b2-ff274d808c3a admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:57:06.940628 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b40094b0-df2a-48c4-bc9b-220227878074 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:57:06.941305 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b40094b0-df2a-48c4-bc9b-220227878074 service cinder] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:57:06.941582 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b40094b0-df2a-48c4-bc9b-220227878074 service cinder] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:57:06.941853 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b40094b0-df2a-48c4-bc9b-220227878074 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:57:06.952571 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b40094b0-df2a-48c4-bc9b-220227878074 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:57:06.954254 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b40094b0-df2a-48c4-bc9b-220227878074 service cinder] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:57:06.966075 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1091/2194] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:57:06 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:57:06.995579 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f51f565-e65d-4aff-8cdd-51fa997afc53 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:57:07.022747 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:57:07.063227 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f51f565-e65d-4aff-8cdd-51fa997afc53 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:57:07.067927 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d56e388-ea45-4b39-acc4-bc8432205040 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:57:07.068566 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d56e388-ea45-4b39-acc4-bc8432205040 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:57:07.068847 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d56e388-ea45-4b39-acc4-bc8432205040 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:57:07.069144 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d56e388-ea45-4b39-acc4-bc8432205040 admin admin] PATH_INFO: `/v3/users/c9f1cd3fee77460d879c55cab26c4a7e` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:57:07.070293 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d56e388-ea45-4b39-acc4-bc8432205040 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c9f1cd3fee77460d879c55cab26c4a7e)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:57:07.072278 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d56e388-ea45-4b39-acc4-bc8432205040 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:57:07.146965 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-5d56e388-ea45-4b39-acc4-bc8432205040 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c9f1cd3fee77460d879c55cab26c4a7e'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:57:07.147615 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-5d56e388-ea45-4b39-acc4-bc8432205040 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c9f1cd3fee77460d879c55cab26c4a7e'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:57:07.159089 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-5d56e388-ea45-4b39-acc4-bc8432205040 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c9f1cd3fee77460d879c55cab26c4a7e'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:57:07.169396 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-5d56e388-ea45-4b39-acc4-bc8432205040 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c9f1cd3fee77460d879c55cab26c4a7e'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:57:07.171219 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-5d56e388-ea45-4b39-acc4-bc8432205040 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c9f1cd3fee77460d879c55cab26c4a7e'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:57:07.193373 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1104/2195] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 00:57:06 2022] DELETE /identity/v3/users/c9f1cd3fee77460d879c55cab26c4a7e => generated 0 bytes in 198 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:57:07.198496 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76365081-a19e-49f3-b639-b3f968ccd193 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:57:07.198909 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76365081-a19e-49f3-b639-b3f968ccd193 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:57:07.199259 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76365081-a19e-49f3-b639-b3f968ccd193 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:57:07.219864 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-76365081-a19e-49f3-b639-b3f968ccd193 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:57:07.221302 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-76365081-a19e-49f3-b639-b3f968ccd193 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:57:07.261673 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:57:07.289356 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1092/2196] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:57:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 28 00:57:07.298889 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d56e388-ea45-4b39-acc4-bc8432205040 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:57:07.300310 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-5d56e388-ea45-4b39-acc4-bc8432205040 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:57:07.368990 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d56e388-ea45-4b39-acc4-bc8432205040 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:57:07.372751 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f6d8c09-965f-41d0-b8a9-1fdfc48140ab service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:57:07.373380 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f6d8c09-965f-41d0-b8a9-1fdfc48140ab service neutron] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:57:07.373653 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f6d8c09-965f-41d0-b8a9-1fdfc48140ab service neutron] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:57:07.373918 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f6d8c09-965f-41d0-b8a9-1fdfc48140ab service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:57:07.383538 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f6d8c09-965f-41d0-b8a9-1fdfc48140ab service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:57:07.385071 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f6d8c09-965f-41d0-b8a9-1fdfc48140ab service neutron] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:57:07.394698 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1105/2197] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:57:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 28 00:57:07.670817 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76365081-a19e-49f3-b639-b3f968ccd193 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:57:07.672352 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-76365081-a19e-49f3-b639-b3f968ccd193 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:57:07.685695 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76365081-a19e-49f3-b639-b3f968ccd193 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:57:07.690825 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-366a6cc9-9a2d-4943-b7df-6c8081a4bc60 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:57:07.691522 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-366a6cc9-9a2d-4943-b7df-6c8081a4bc60 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:57:07.691881 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-366a6cc9-9a2d-4943-b7df-6c8081a4bc60 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:57:07.692160 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-366a6cc9-9a2d-4943-b7df-6c8081a4bc60 admin admin] PATH_INFO: `/v3/projects/4b4cf8ca6866462fb1eaa31c3fe9341b` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:57:07.693353 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-366a6cc9-9a2d-4943-b7df-6c8081a4bc60 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4b4cf8ca6866462fb1eaa31c3fe9341b)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:57:07.695307 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-366a6cc9-9a2d-4943-b7df-6c8081a4bc60 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:57:07.746381 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-366a6cc9-9a2d-4943-b7df-6c8081a4bc60 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4b4cf8ca6866462fb1eaa31c3fe9341b', 'request_id': 'req-366a6cc9-9a2d-4943-b7df-6c8081a4bc60'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:57:07.752716 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-366a6cc9-9a2d-4943-b7df-6c8081a4bc60 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4b4cf8ca6866462fb1eaa31c3fe9341b', 'request_id': 'req-366a6cc9-9a2d-4943-b7df-6c8081a4bc60'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:57:07.759384 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-366a6cc9-9a2d-4943-b7df-6c8081a4bc60 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4b4cf8ca6866462fb1eaa31c3fe9341b', 'request_id': 'req-366a6cc9-9a2d-4943-b7df-6c8081a4bc60'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:57:07.774226 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-366a6cc9-9a2d-4943-b7df-6c8081a4bc60 admin admin] The token cache is being invalidate because project 4b4cf8ca6866462fb1eaa31c3fe9341b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81507) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 00:57:07.774514 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-366a6cc9-9a2d-4943-b7df-6c8081a4bc60 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:57:07.775115 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-366a6cc9-9a2d-4943-b7df-6c8081a4bc60 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:57:07.776623 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1093/2198] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 00:57:07 2022] DELETE /identity/v3/projects/4b4cf8ca6866462fb1eaa31c3fe9341b => generated 0 bytes in 106 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:57:07.781752 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21906a3e-d448-4a1c-b0a6-95b1fc5add5b None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:57:07.782251 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21906a3e-d448-4a1c-b0a6-95b1fc5add5b None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:57:07.782251 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21906a3e-d448-4a1c-b0a6-95b1fc5add5b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:57:07.803834 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-21906a3e-d448-4a1c-b0a6-95b1fc5add5b None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:57:07.805111 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-21906a3e-d448-4a1c-b0a6-95b1fc5add5b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:57:07.848829 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:57:07.868822 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1106/2199] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:57:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:57:07.876744 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9bb5e93-475e-4a06-9937-967f017dbfa9 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:57:07.877151 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9bb5e93-475e-4a06-9937-967f017dbfa9 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:57:07.877409 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9bb5e93-475e-4a06-9937-967f017dbfa9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:57:07.898278 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-e9bb5e93-475e-4a06-9937-967f017dbfa9 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:57:07.899491 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-e9bb5e93-475e-4a06-9937-967f017dbfa9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:57:07.908968 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1094/2200] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:57:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:57:07.913786 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21906a3e-d448-4a1c-b0a6-95b1fc5add5b None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:57:07.924203 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21906a3e-d448-4a1c-b0a6-95b1fc5add5b None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:57:07.930176 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-815bc112-2065-4190-9873-58aaba46885c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:57:07.930780 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-815bc112-2065-4190-9873-58aaba46885c admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:57:07.931057 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-815bc112-2065-4190-9873-58aaba46885c admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:57:07.931324 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-815bc112-2065-4190-9873-58aaba46885c admin admin] PATH_INFO: `/v3/domains` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:57:07.931973 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-815bc112-2065-4190-9873-58aaba46885c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:57:07.932267 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-815bc112-2065-4190-9873-58aaba46885c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:57:07.933985 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-815bc112-2065-4190-9873-58aaba46885c admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:57:07.946295 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1107/2201] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Aug 28 00:57:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:57:07.951143 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9bb5e93-475e-4a06-9937-967f017dbfa9 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:57:07.960659 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9bb5e93-475e-4a06-9937-967f017dbfa9 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:57:07.965921 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54c2ac80-49f9-40e0-a889-25e366a1da1c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:57:07.966514 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c2ac80-49f9-40e0-a889-25e366a1da1c admin admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:57:07.966792 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c2ac80-49f9-40e0-a889-25e366a1da1c admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:57:07.967056 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c2ac80-49f9-40e0-a889-25e366a1da1c admin admin] PATH_INFO: `/v3/projects` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:57:07.967682 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54c2ac80-49f9-40e0-a889-25e366a1da1c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:57:07.969621 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54c2ac80-49f9-40e0-a889-25e366a1da1c admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:57:07.998232 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1095/2202] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 00:57:07 2022] POST /identity/v3/projects => generated 379 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:57:08.002952 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-815bc112-2065-4190-9873-58aaba46885c admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:57:08.013383 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-815bc112-2065-4190-9873-58aaba46885c admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:57:08.074042 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1cf6c3f-0662-42c0-8358-38243de03c6e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:57:08.074660 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1cf6c3f-0662-42c0-8358-38243de03c6e admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:57:08.074939 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1cf6c3f-0662-42c0-8358-38243de03c6e admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:57:08.075182 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1cf6c3f-0662-42c0-8358-38243de03c6e admin admin] PATH_INFO: `/v3/users` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:57:08.075813 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1cf6c3f-0662-42c0-8358-38243de03c6e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:57:08.077796 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1cf6c3f-0662-42c0-8358-38243de03c6e admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:57:08.102311 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1108/2203] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 00:57:08 2022] POST /identity/v3/users => generated 290 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:57:08.106552 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54c2ac80-49f9-40e0-a889-25e366a1da1c admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:57:08.118432 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54c2ac80-49f9-40e0-a889-25e366a1da1c admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:57:08.123602 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30f6720f-69cd-4f16-8a34-ecfe9a54362b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:57:08.124212 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30f6720f-69cd-4f16-8a34-ecfe9a54362b admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:57:08.124726 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30f6720f-69cd-4f16-8a34-ecfe9a54362b admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:57:08.125031 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30f6720f-69cd-4f16-8a34-ecfe9a54362b admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:57:08.125773 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30f6720f-69cd-4f16-8a34-ecfe9a54362b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:57:08.127411 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30f6720f-69cd-4f16-8a34-ecfe9a54362b admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:57:08.137857 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1096/2204] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:57:08 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 00:57:08.142164 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1cf6c3f-0662-42c0-8358-38243de03c6e admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:57:08.152731 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1cf6c3f-0662-42c0-8358-38243de03c6e admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:57:08.158584 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22194851-eeea-410c-807c-9367a54fcd2e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:57:08.159191 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22194851-eeea-410c-807c-9367a54fcd2e admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:57:08.159487 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22194851-eeea-410c-807c-9367a54fcd2e admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:57:08.159742 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22194851-eeea-410c-807c-9367a54fcd2e admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:57:08.160385 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22194851-eeea-410c-807c-9367a54fcd2e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:57:08.162068 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22194851-eeea-410c-807c-9367a54fcd2e admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:57:08.173422 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1109/2205] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:57:08 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 00:57:08.177781 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30f6720f-69cd-4f16-8a34-ecfe9a54362b admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:57:08.188508 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30f6720f-69cd-4f16-8a34-ecfe9a54362b admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:57:08.193464 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55601940-3150-45c0-995e-2b6901a81a06 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:57:08.194128 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55601940-3150-45c0-995e-2b6901a81a06 admin admin] REQUEST_METHOD: `PUT` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:57:08.194408 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55601940-3150-45c0-995e-2b6901a81a06 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:57:08.194679 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55601940-3150-45c0-995e-2b6901a81a06 admin admin] PATH_INFO: `/v3/projects/287c855ef8e241978a4d1e7cb3c7a16f/users/cac1ab72dcee40b19bcff4dd5a75d82d/roles/0487ef7e943e4701915cb8ce0f2899d7` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:57:08.211193 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55601940-3150-45c0-995e-2b6901a81a06 admin admin] RBAC: Authorizing `identity:create_grant(project_id=287c855ef8e241978a4d1e7cb3c7a16f, user_id=cac1ab72dcee40b19bcff4dd5a75d82d, role_id=0487ef7e943e4701915cb8ce0f2899d7)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:57:08.213980 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55601940-3150-45c0-995e-2b6901a81a06 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:57:08.229031 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1097/2206] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 00:57:08 2022] PUT /identity/v3/projects/287c855ef8e241978a4d1e7cb3c7a16f/users/cac1ab72dcee40b19bcff4dd5a75d82d/roles/0487ef7e943e4701915cb8ce0f2899d7 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:57:08.234409 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da09df2b-db19-4fd7-b50f-7d86b6abb654 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:57:08.234725 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da09df2b-db19-4fd7-b50f-7d86b6abb654 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:57:08.234999 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da09df2b-db19-4fd7-b50f-7d86b6abb654 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:57:08.254720 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-da09df2b-db19-4fd7-b50f-7d86b6abb654 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:57:08.255971 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-da09df2b-db19-4fd7-b50f-7d86b6abb654 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:57:08.319558 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1110/2207] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:57:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:57:08.330504 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55601940-3150-45c0-995e-2b6901a81a06 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:57:08.332541 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-55601940-3150-45c0-995e-2b6901a81a06 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:57:08.404656 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55601940-3150-45c0-995e-2b6901a81a06 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:57:08.409283 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0852805b-6985-45f5-8530-7e7f7301a34a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:57:08.409900 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0852805b-6985-45f5-8530-7e7f7301a34a service neutron] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:57:08.410180 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0852805b-6985-45f5-8530-7e7f7301a34a service neutron] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:57:08.410447 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0852805b-6985-45f5-8530-7e7f7301a34a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:57:08.422736 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0852805b-6985-45f5-8530-7e7f7301a34a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:57:08.424602 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0852805b-6985-45f5-8530-7e7f7301a34a service neutron] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:57:08.435889 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1098/2208] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:57:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:57:13.822369 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ef4b395-ba8e-48b0-ac0a-34d8bb6e7afd None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:57:13.823252 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ef4b395-ba8e-48b0-ac0a-34d8bb6e7afd None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:57:13.823252 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ef4b395-ba8e-48b0-ac0a-34d8bb6e7afd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:57:13.889101 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-7ef4b395-ba8e-48b0-ac0a-34d8bb6e7afd None None] MFA Rules not processed for user `cac1ab72dcee40b19bcff4dd5a75d82d`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:57:13.893068 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-7ef4b395-ba8e-48b0-ac0a-34d8bb6e7afd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:57:13.974442 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:57:14.000894 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1111/2209] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:57:13 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:57:14.005454 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0852805b-6985-45f5-8530-7e7f7301a34a service neutron] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:57:14.010865 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:57:14.018578 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0852805b-6985-45f5-8530-7e7f7301a34a service neutron] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:57:14.024373 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cf87cf2-3c2a-474c-8795-04da349409ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:57:14.025101 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cf87cf2-3c2a-474c-8795-04da349409ca admin admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:57:14.025787 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cf87cf2-3c2a-474c-8795-04da349409ca admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:57:14.026132 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cf87cf2-3c2a-474c-8795-04da349409ca admin admin] PATH_INFO: `/v3/projects` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:57:14.027409 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cf87cf2-3c2a-474c-8795-04da349409ca admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:57:14.029404 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cf87cf2-3c2a-474c-8795-04da349409ca admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:57:14.058424 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1099/2210] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 00:57:14 2022] POST /identity/v3/projects => generated 379 bytes in 53 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 28 00:57:14.062525 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ef4b395-ba8e-48b0-ac0a-34d8bb6e7afd None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:57:14.075103 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ef4b395-ba8e-48b0-ac0a-34d8bb6e7afd None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:57:14.136636 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dda54ed-6a0b-4cb8-9f3b-d9eced4e8a00 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:57:14.137280 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dda54ed-6a0b-4cb8-9f3b-d9eced4e8a00 admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:57:14.137553 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dda54ed-6a0b-4cb8-9f3b-d9eced4e8a00 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:57:14.137818 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dda54ed-6a0b-4cb8-9f3b-d9eced4e8a00 admin admin] PATH_INFO: `/v3/users` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:57:14.138462 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dda54ed-6a0b-4cb8-9f3b-d9eced4e8a00 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:57:14.140393 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dda54ed-6a0b-4cb8-9f3b-d9eced4e8a00 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:57:14.165939 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1112/2211] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 00:57:14 2022] POST /identity/v3/users => generated 296 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:57:14.170221 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cf87cf2-3c2a-474c-8795-04da349409ca admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:57:14.181535 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cf87cf2-3c2a-474c-8795-04da349409ca admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:57:14.189325 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fff44f63-8782-49ce-b911-3a5ca9d0ff6e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:57:14.190038 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff44f63-8782-49ce-b911-3a5ca9d0ff6e admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:57:14.190324 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff44f63-8782-49ce-b911-3a5ca9d0ff6e admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:57:14.190791 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff44f63-8782-49ce-b911-3a5ca9d0ff6e admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:57:14.191908 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fff44f63-8782-49ce-b911-3a5ca9d0ff6e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:57:14.195371 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fff44f63-8782-49ce-b911-3a5ca9d0ff6e admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:57:14.209013 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1100/2212] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:57:14 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 00:57:14.212491 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dda54ed-6a0b-4cb8-9f3b-d9eced4e8a00 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:57:14.224390 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dda54ed-6a0b-4cb8-9f3b-d9eced4e8a00 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:57:14.232817 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9612cfd-ba52-4055-a9df-2b6d9d6b9957 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:57:14.233564 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9612cfd-ba52-4055-a9df-2b6d9d6b9957 admin admin] REQUEST_METHOD: `PUT` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:57:14.233937 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9612cfd-ba52-4055-a9df-2b6d9d6b9957 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:57:14.234220 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9612cfd-ba52-4055-a9df-2b6d9d6b9957 admin admin] PATH_INFO: `/v3/projects/7a69dc53c1a648c7a764c8ed1a1aeac7/users/4cac7d604715464d85da907d1dfc3482/roles/f08d6043261747e3b4763c744eef7dfb` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:57:14.252106 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9612cfd-ba52-4055-a9df-2b6d9d6b9957 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7a69dc53c1a648c7a764c8ed1a1aeac7, user_id=4cac7d604715464d85da907d1dfc3482, role_id=f08d6043261747e3b4763c744eef7dfb)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:57:14.254986 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9612cfd-ba52-4055-a9df-2b6d9d6b9957 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:57:14.277804 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1113/2213] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 00:57:14 2022] PUT /identity/v3/projects/7a69dc53c1a648c7a764c8ed1a1aeac7/users/4cac7d604715464d85da907d1dfc3482/roles/f08d6043261747e3b4763c744eef7dfb => generated 0 bytes in 66 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:57:14.282793 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fff44f63-8782-49ce-b911-3a5ca9d0ff6e admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:57:14.294013 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fff44f63-8782-49ce-b911-3a5ca9d0ff6e admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:57:14.358040 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebc9cb7f-c1b6-4f08-a287-8b2fea011148 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:57:14.358710 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebc9cb7f-c1b6-4f08-a287-8b2fea011148 admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:57:14.360269 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebc9cb7f-c1b6-4f08-a287-8b2fea011148 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:57:14.360269 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebc9cb7f-c1b6-4f08-a287-8b2fea011148 admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:57:14.360269 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebc9cb7f-c1b6-4f08-a287-8b2fea011148 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:57:14.361872 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebc9cb7f-c1b6-4f08-a287-8b2fea011148 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:57:14.372758 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1101/2214] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:57:14 2022] GET /identity/v3/roles => generated 1420 bytes in 91 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 00:57:14.377648 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9612cfd-ba52-4055-a9df-2b6d9d6b9957 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:57:14.390263 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9612cfd-ba52-4055-a9df-2b6d9d6b9957 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:57:14.395621 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8acad5c-fb12-4c63-872f-b0e028ba2107 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:57:14.396330 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8acad5c-fb12-4c63-872f-b0e028ba2107 admin admin] REQUEST_METHOD: `PUT` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:57:14.396610 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8acad5c-fb12-4c63-872f-b0e028ba2107 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:57:14.396955 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8acad5c-fb12-4c63-872f-b0e028ba2107 admin admin] PATH_INFO: `/v3/projects/7a69dc53c1a648c7a764c8ed1a1aeac7/users/4cac7d604715464d85da907d1dfc3482/roles/f08d6043261747e3b4763c744eef7dfb` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:57:14.398930 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8acad5c-fb12-4c63-872f-b0e028ba2107 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7a69dc53c1a648c7a764c8ed1a1aeac7, user_id=4cac7d604715464d85da907d1dfc3482, role_id=f08d6043261747e3b4763c744eef7dfb)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:57:14.401660 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8acad5c-fb12-4c63-872f-b0e028ba2107 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:57:14.414398 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1114/2215] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 00:57:14 2022] PUT /identity/v3/projects/7a69dc53c1a648c7a764c8ed1a1aeac7/users/4cac7d604715464d85da907d1dfc3482/roles/f08d6043261747e3b4763c744eef7dfb => generated 0 bytes in 37 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:57:20.176922 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f275123e-6434-4f1d-bc3a-148d90e6f812 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:57:20.177814 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f275123e-6434-4f1d-bc3a-148d90e6f812 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:57:20.177978 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f275123e-6434-4f1d-bc3a-148d90e6f812 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:57:20.226239 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-f275123e-6434-4f1d-bc3a-148d90e6f812 None None] MFA Rules not processed for user `4cac7d604715464d85da907d1dfc3482`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:57:20.227890 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-f275123e-6434-4f1d-bc3a-148d90e6f812 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:57:20.330615 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1102/2216] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:57:20 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 28 00:57:20.350563 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8acad5c-fb12-4c63-872f-b0e028ba2107 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:57:20.353143 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-f8acad5c-fb12-4c63-872f-b0e028ba2107 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:57:20.474926 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8acad5c-fb12-4c63-872f-b0e028ba2107 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:57:20.490869 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd2e1e72-062e-45f7-acd2-7a0453569409 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '168b07df50c24593ba73dbde5b037aed', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:57:20.491671 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd2e1e72-062e-45f7-acd2-7a0453569409 service nova] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:57:20.492076 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd2e1e72-062e-45f7-acd2-7a0453569409 service nova] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:57:20.493238 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd2e1e72-062e-45f7-acd2-7a0453569409 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:57:20.558291 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:57:20.656612 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd2e1e72-062e-45f7-acd2-7a0453569409 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:57:20.660485 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd2e1e72-062e-45f7-acd2-7a0453569409 service nova] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:57:20.678606 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1115/2217] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:57:20 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 328 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 28 00:57:23.910439 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42b9e1ed-2816-4317-81d2-52db970e6799 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:57:23.910993 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42b9e1ed-2816-4317-81d2-52db970e6799 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:57:23.910993 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42b9e1ed-2816-4317-81d2-52db970e6799 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:57:23.935879 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-42b9e1ed-2816-4317-81d2-52db970e6799 None None] MFA Rules not processed for user `59b04859082547149bf2776aefa63b43`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:57:23.939979 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-42b9e1ed-2816-4317-81d2-52db970e6799 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:57:24.010206 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:57:24.026557 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1103/2218] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:57:23 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:57:26.607541 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db7c290d-8d06-48b6-b71d-5fb2f48b2dcd None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:57:26.608482 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db7c290d-8d06-48b6-b71d-5fb2f48b2dcd None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:57:26.608482 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db7c290d-8d06-48b6-b71d-5fb2f48b2dcd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:57:26.636117 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-db7c290d-8d06-48b6-b71d-5fb2f48b2dcd None None] MFA Rules not processed for user `1ab2fcb3759045c7a076e5c8222a421d`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:57:26.637594 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-db7c290d-8d06-48b6-b71d-5fb2f48b2dcd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:57:26.731931 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1116/2219] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:57:26 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 28 00:57:34.019259 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=2 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:57:36.718358 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=39 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:57:47.628946 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5af3ca5-6863-4fdd-9c02-88c1e086dd03 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:57:47.629710 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5af3ca5-6863-4fdd-9c02-88c1e086dd03 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:57:47.629710 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5af3ca5-6863-4fdd-9c02-88c1e086dd03 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:57:47.651362 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-e5af3ca5-6863-4fdd-9c02-88c1e086dd03 None None] MFA Rules not processed for user `cac1ab72dcee40b19bcff4dd5a75d82d`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:57:47.652946 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-e5af3ca5-6863-4fdd-9c02-88c1e086dd03 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:57:47.662129 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1104/2220] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:57:47 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:57:57.645912 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=3 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:57:59.778573 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5219882a-d8b4-45b5-8a1c-eb5725e19265 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:57:59.779548 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5219882a-d8b4-45b5-8a1c-eb5725e19265 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:57:59.779548 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5219882a-d8b4-45b5-8a1c-eb5725e19265 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:57:59.800900 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-5219882a-d8b4-45b5-8a1c-eb5725e19265 None None] MFA Rules not processed for user `eede08f020e04282bb994b855e371433`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:57:59.802351 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-5219882a-d8b4-45b5-8a1c-eb5725e19265 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:57:59.906423 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1117/2221] 10.222.0.42 () {58 vars in 1009 bytes} [Sun Aug 28 00:57:59 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:57:59.971483 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5af3ca5-6863-4fdd-9c02-88c1e086dd03 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:57:59.973741 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-e5af3ca5-6863-4fdd-9c02-88c1e086dd03 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:58:00.081791 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5af3ca5-6863-4fdd-9c02-88c1e086dd03 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:58:00.088575 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3628b96-e2ce-4519-ab4e-f38c042f65a8 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac4c5584582f4f55a7085af4cbbabf9b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:58:00.089766 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3628b96-e2ce-4519-ab4e-f38c042f65a8 service swift] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:58:00.090220 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3628b96-e2ce-4519-ab4e-f38c042f65a8 service swift] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:58:00.090668 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3628b96-e2ce-4519-ab4e-f38c042f65a8 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:58:00.102856 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3628b96-e2ce-4519-ab4e-f38c042f65a8 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:58:00.105263 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3628b96-e2ce-4519-ab4e-f38c042f65a8 service swift] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:58:00.121154 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1105/2222] 10.222.0.42 () {60 vars in 1384 bytes} [Sun Aug 28 00:57:59 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 150 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 28 00:58:09.899112 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=25 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:58:10.117316 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=32 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:58:16.866731 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5219882a-d8b4-45b5-8a1c-eb5725e19265 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:58:17.001518 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5219882a-d8b4-45b5-8a1c-eb5725e19265 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:58:17.014176 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88d13971-2953-435b-b52c-9a25d9f4bc0e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:58:17.015430 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88d13971-2953-435b-b52c-9a25d9f4bc0e service neutron] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:58:17.017486 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88d13971-2953-435b-b52c-9a25d9f4bc0e service neutron] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:58:17.017887 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88d13971-2953-435b-b52c-9a25d9f4bc0e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:58:17.034204 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88d13971-2953-435b-b52c-9a25d9f4bc0e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:58:17.039463 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88d13971-2953-435b-b52c-9a25d9f4bc0e service neutron] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:58:17.065061 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1118/2223] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:58:16 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 199 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:58:21.702248 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aae15ca2-7230-46e7-810e-48aa0141425b None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:58:21.703193 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aae15ca2-7230-46e7-810e-48aa0141425b None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:58:21.703193 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aae15ca2-7230-46e7-810e-48aa0141425b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:58:21.733220 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-aae15ca2-7230-46e7-810e-48aa0141425b None None] MFA Rules not processed for user `eede08f020e04282bb994b855e371433`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:58:21.735034 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-aae15ca2-7230-46e7-810e-48aa0141425b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:58:21.747023 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1106/2224] 10.222.0.42 () {58 vars in 967 bytes} [Sun Aug 28 00:58:21 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 46 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:58:21.764600 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88d13971-2953-435b-b52c-9a25d9f4bc0e service neutron] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:58:21.785779 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88d13971-2953-435b-b52c-9a25d9f4bc0e service neutron] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:58:21.794636 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a04642f-aeb6-4709-884d-c17215e86fa4 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac4c5584582f4f55a7085af4cbbabf9b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:58:21.795694 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a04642f-aeb6-4709-884d-c17215e86fa4 service swift] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:58:21.796190 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a04642f-aeb6-4709-884d-c17215e86fa4 service swift] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:58:21.796705 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a04642f-aeb6-4709-884d-c17215e86fa4 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:58:21.812880 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a04642f-aeb6-4709-884d-c17215e86fa4 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:58:21.815835 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a04642f-aeb6-4709-884d-c17215e86fa4 service swift] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:58:21.828533 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1119/2225] 10.222.0.42 () {60 vars in 1384 bytes} [Sun Aug 28 00:58:21 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 65 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 28 00:58:24.193352 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aae15ca2-7230-46e7-810e-48aa0141425b None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:58:24.196039 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-aae15ca2-7230-46e7-810e-48aa0141425b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:58:24.298736 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aae15ca2-7230-46e7-810e-48aa0141425b None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:58:24.306883 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0343a4d-b844-43cf-ae99-9c1b8d3f831f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:58:24.307989 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0343a4d-b844-43cf-ae99-9c1b8d3f831f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:58:24.308481 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0343a4d-b844-43cf-ae99-9c1b8d3f831f admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:58:24.308981 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0343a4d-b844-43cf-ae99-9c1b8d3f831f admin admin] PATH_INFO: `/v3/users/59b04859082547149bf2776aefa63b43` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:58:24.311310 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0343a4d-b844-43cf-ae99-9c1b8d3f831f admin admin] RBAC: Authorizing `identity:delete_user(user_id=59b04859082547149bf2776aefa63b43)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:58:24.315147 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0343a4d-b844-43cf-ae99-9c1b8d3f831f admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:58:24.389592 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-f0343a4d-b844-43cf-ae99-9c1b8d3f831f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '59b04859082547149bf2776aefa63b43'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:58:24.419386 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-f0343a4d-b844-43cf-ae99-9c1b8d3f831f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '59b04859082547149bf2776aefa63b43'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:58:24.432839 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-f0343a4d-b844-43cf-ae99-9c1b8d3f831f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '59b04859082547149bf2776aefa63b43'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:58:24.447208 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-f0343a4d-b844-43cf-ae99-9c1b8d3f831f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '59b04859082547149bf2776aefa63b43'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:58:24.448084 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-f0343a4d-b844-43cf-ae99-9c1b8d3f831f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '59b04859082547149bf2776aefa63b43'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:58:24.455917 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1107/2226] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 00:58:24 2022] DELETE /identity/v3/users/59b04859082547149bf2776aefa63b43 => generated 0 bytes in 264 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:58:24.752133 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a04642f-aeb6-4709-884d-c17215e86fa4 service swift] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:58:24.771560 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a04642f-aeb6-4709-884d-c17215e86fa4 service swift] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:58:24.853773 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b57502df-ecc6-4072-88be-78fb53c8f8cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:58:24.854478 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b57502df-ecc6-4072-88be-78fb53c8f8cd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:58:24.854787 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b57502df-ecc6-4072-88be-78fb53c8f8cd admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:58:24.855207 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b57502df-ecc6-4072-88be-78fb53c8f8cd admin admin] PATH_INFO: `/v3/projects/315d6d72ee5844edb203b740ab2d08ce` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:58:24.856572 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b57502df-ecc6-4072-88be-78fb53c8f8cd admin admin] RBAC: Authorizing `identity:delete_project(project_id=315d6d72ee5844edb203b740ab2d08ce)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:58:24.858893 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b57502df-ecc6-4072-88be-78fb53c8f8cd admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:58:24.925495 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-b57502df-ecc6-4072-88be-78fb53c8f8cd admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '315d6d72ee5844edb203b740ab2d08ce', 'request_id': 'req-b57502df-ecc6-4072-88be-78fb53c8f8cd'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:58:24.937523 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-b57502df-ecc6-4072-88be-78fb53c8f8cd admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '315d6d72ee5844edb203b740ab2d08ce', 'request_id': 'req-b57502df-ecc6-4072-88be-78fb53c8f8cd'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:58:24.952669 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-b57502df-ecc6-4072-88be-78fb53c8f8cd admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '315d6d72ee5844edb203b740ab2d08ce', 'request_id': 'req-b57502df-ecc6-4072-88be-78fb53c8f8cd'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:58:24.990387 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-b57502df-ecc6-4072-88be-78fb53c8f8cd admin admin] The token cache is being invalidate because project 315d6d72ee5844edb203b740ab2d08ce was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81508) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 00:58:24.991009 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-b57502df-ecc6-4072-88be-78fb53c8f8cd admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:58:24.992546 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-b57502df-ecc6-4072-88be-78fb53c8f8cd admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:58:24.995783 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1120/2227] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 00:58:24 2022] DELETE /identity/v3/projects/315d6d72ee5844edb203b740ab2d08ce => generated 0 bytes in 245 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:58:25.001007 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28223e1a-a6c3-4f89-9bdb-b60536c60bee None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:58:25.001360 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28223e1a-a6c3-4f89-9bdb-b60536c60bee None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:58:25.001638 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28223e1a-a6c3-4f89-9bdb-b60536c60bee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:58:25.031730 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-28223e1a-a6c3-4f89-9bdb-b60536c60bee None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:58:25.033211 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-28223e1a-a6c3-4f89-9bdb-b60536c60bee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:58:25.109726 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1108/2228] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:58:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 28 00:58:25.124645 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01563d95-8900-4437-b515-5b92c7dc86e4 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:58:25.125626 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01563d95-8900-4437-b515-5b92c7dc86e4 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:58:25.127679 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01563d95-8900-4437-b515-5b92c7dc86e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:58:25.161522 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-01563d95-8900-4437-b515-5b92c7dc86e4 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:58:25.163666 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-01563d95-8900-4437-b515-5b92c7dc86e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:58:25.178440 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1121/2229] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:58:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 56 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:58:25.182590 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28223e1a-a6c3-4f89-9bdb-b60536c60bee None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:58:25.194642 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28223e1a-a6c3-4f89-9bdb-b60536c60bee None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:58:25.200997 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7ff00b2-5a25-4b11-bc80-8f5890ff8fa7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:58:25.201671 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7ff00b2-5a25-4b11-bc80-8f5890ff8fa7 admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:58:25.201943 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7ff00b2-5a25-4b11-bc80-8f5890ff8fa7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:58:25.202240 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7ff00b2-5a25-4b11-bc80-8f5890ff8fa7 admin admin] PATH_INFO: `/v3/domains` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:58:25.202915 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7ff00b2-5a25-4b11-bc80-8f5890ff8fa7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:58:25.203216 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7ff00b2-5a25-4b11-bc80-8f5890ff8fa7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:58:25.204914 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7ff00b2-5a25-4b11-bc80-8f5890ff8fa7 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:58:25.215488 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:58:25.220049 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1109/2230] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Aug 28 00:58:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:58:25.227047 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01563d95-8900-4437-b515-5b92c7dc86e4 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:58:25.238906 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01563d95-8900-4437-b515-5b92c7dc86e4 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:58:25.245113 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b354506d-77d8-4b6b-b7df-d38c60113e8d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:58:25.245772 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b354506d-77d8-4b6b-b7df-d38c60113e8d admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:58:25.246136 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b354506d-77d8-4b6b-b7df-d38c60113e8d admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:58:25.246438 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b354506d-77d8-4b6b-b7df-d38c60113e8d admin admin] PATH_INFO: `/v3/projects` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:58:25.247210 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b354506d-77d8-4b6b-b7df-d38c60113e8d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:58:25.249457 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b354506d-77d8-4b6b-b7df-d38c60113e8d admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:58:25.264682 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:58:25.293680 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1122/2231] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 00:58:25 2022] POST /identity/v3/projects => generated 385 bytes in 67 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:58:25.300038 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7ff00b2-5a25-4b11-bc80-8f5890ff8fa7 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:58:25.319291 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7ff00b2-5a25-4b11-bc80-8f5890ff8fa7 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:58:25.405413 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13ca1ad1-ccfe-4219-b19b-b73535c40c85 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:58:25.406459 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13ca1ad1-ccfe-4219-b19b-b73535c40c85 admin admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:58:25.406972 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13ca1ad1-ccfe-4219-b19b-b73535c40c85 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:58:25.407398 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13ca1ad1-ccfe-4219-b19b-b73535c40c85 admin admin] PATH_INFO: `/v3/users` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:58:25.408099 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13ca1ad1-ccfe-4219-b19b-b73535c40c85 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:58:25.411257 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13ca1ad1-ccfe-4219-b19b-b73535c40c85 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:58:25.447622 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1110/2232] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 00:58:25 2022] POST /identity/v3/users => generated 293 bytes in 148 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:58:25.452936 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b354506d-77d8-4b6b-b7df-d38c60113e8d admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:58:25.471525 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b354506d-77d8-4b6b-b7df-d38c60113e8d admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:58:25.480241 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65de0ff8-576c-4981-88c9-26aa6623802c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:58:25.481458 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65de0ff8-576c-4981-88c9-26aa6623802c admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:58:25.482047 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65de0ff8-576c-4981-88c9-26aa6623802c admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:58:25.482609 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65de0ff8-576c-4981-88c9-26aa6623802c admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:58:25.483917 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65de0ff8-576c-4981-88c9-26aa6623802c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:58:25.487313 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65de0ff8-576c-4981-88c9-26aa6623802c admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:58:25.507772 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1123/2233] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:58:25 2022] GET /identity/v3/roles => generated 1420 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 00:58:25.514764 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13ca1ad1-ccfe-4219-b19b-b73535c40c85 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:58:25.527459 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13ca1ad1-ccfe-4219-b19b-b73535c40c85 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:58:25.533621 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3bdde92-0cc9-4881-8a5d-8abdf5fc2846 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:58:25.534359 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3bdde92-0cc9-4881-8a5d-8abdf5fc2846 admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:58:25.534698 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3bdde92-0cc9-4881-8a5d-8abdf5fc2846 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:58:25.534979 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3bdde92-0cc9-4881-8a5d-8abdf5fc2846 admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:58:25.535760 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3bdde92-0cc9-4881-8a5d-8abdf5fc2846 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:58:25.537722 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3bdde92-0cc9-4881-8a5d-8abdf5fc2846 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:58:25.550564 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1111/2234] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:58:25 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 00:58:25.555348 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65de0ff8-576c-4981-88c9-26aa6623802c admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:58:25.566648 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65de0ff8-576c-4981-88c9-26aa6623802c admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:58:25.573619 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1169516b-5536-4243-ab54-fc609aa7005e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:58:25.574319 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1169516b-5536-4243-ab54-fc609aa7005e admin admin] REQUEST_METHOD: `PUT` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:58:25.574613 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1169516b-5536-4243-ab54-fc609aa7005e admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:58:25.574887 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1169516b-5536-4243-ab54-fc609aa7005e admin admin] PATH_INFO: `/v3/projects/5feae4c5cde347b5a647c028f134f80d/users/03923cd0847f48d783361a92bdde030c/roles/0487ef7e943e4701915cb8ce0f2899d7` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:58:25.600904 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1169516b-5536-4243-ab54-fc609aa7005e admin admin] RBAC: Authorizing `identity:create_grant(project_id=5feae4c5cde347b5a647c028f134f80d, user_id=03923cd0847f48d783361a92bdde030c, role_id=0487ef7e943e4701915cb8ce0f2899d7)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:58:25.604179 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1169516b-5536-4243-ab54-fc609aa7005e admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:58:25.622690 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1124/2235] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 00:58:25 2022] PUT /identity/v3/projects/5feae4c5cde347b5a647c028f134f80d/users/03923cd0847f48d783361a92bdde030c/roles/0487ef7e943e4701915cb8ce0f2899d7 => generated 0 bytes in 68 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:58:25.635725 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9fa9976-e02f-4858-9b41-d49855511637 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:58:25.636356 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9fa9976-e02f-4858-9b41-d49855511637 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:58:25.636736 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9fa9976-e02f-4858-9b41-d49855511637 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:58:25.676491 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-d9fa9976-e02f-4858-9b41-d49855511637 None None] MFA Rules not processed for user `03923cd0847f48d783361a92bdde030c`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:58:25.677906 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-d9fa9976-e02f-4858-9b41-d49855511637 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:58:25.797035 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1112/2236] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:58:25 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:58:25.817529 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1169516b-5536-4243-ab54-fc609aa7005e admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:58:25.820308 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-1169516b-5536-4243-ab54-fc609aa7005e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:58:25.914692 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1169516b-5536-4243-ab54-fc609aa7005e admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:58:25.918886 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3345664f-d461-4432-964e-5764c81faf3f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:58:25.919526 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3345664f-d461-4432-964e-5764c81faf3f service cinder] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:58:25.919809 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3345664f-d461-4432-964e-5764c81faf3f service cinder] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:58:25.920077 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3345664f-d461-4432-964e-5764c81faf3f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:58:25.929918 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3345664f-d461-4432-964e-5764c81faf3f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:58:25.931964 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3345664f-d461-4432-964e-5764c81faf3f service cinder] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:58:25.944747 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1125/2237] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:58:25 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 129 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 28 00:58:26.965104 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bbbcb94-c74e-4848-ac73-96ca4290846b None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:58:26.965856 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bbbcb94-c74e-4848-ac73-96ca4290846b None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:58:26.965993 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bbbcb94-c74e-4848-ac73-96ca4290846b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:58:26.996596 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-3bbbcb94-c74e-4848-ac73-96ca4290846b None None] MFA Rules not processed for user `d209174d0db04b3584556824c6d0a3d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:58:26.997944 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-3bbbcb94-c74e-4848-ac73-96ca4290846b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:58:27.060869 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1113/2238] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:58:26 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:58:28.537177 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3345664f-d461-4432-964e-5764c81faf3f service cinder] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:58:28.538836 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-3345664f-d461-4432-964e-5764c81faf3f service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:58:28.626615 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3345664f-d461-4432-964e-5764c81faf3f service cinder] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:58:28.632290 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c9cbbed-3c8e-4b1e-8d9a-b0403e6acf8d service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2a3834fa6894dada3291e4781bbd553', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:58:28.632920 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c9cbbed-3c8e-4b1e-8d9a-b0403e6acf8d service placement] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:58:28.633217 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c9cbbed-3c8e-4b1e-8d9a-b0403e6acf8d service placement] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:58:28.633496 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c9cbbed-3c8e-4b1e-8d9a-b0403e6acf8d service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:58:28.635441 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-5c9cbbed-3c8e-4b1e-8d9a-b0403e6acf8d service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:58:28.648087 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c9cbbed-3c8e-4b1e-8d9a-b0403e6acf8d service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:58:28.650041 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c9cbbed-3c8e-4b1e-8d9a-b0403e6acf8d service placement] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:58:28.661966 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1126/2239] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:58:28 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 125 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:58:28.774888 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bbbcb94-c74e-4848-ac73-96ca4290846b None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:58:28.802098 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:58:28.924468 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bbbcb94-c74e-4848-ac73-96ca4290846b None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:58:28.929906 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aded15bb-391a-44da-b5dc-2e9ff016cfc8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:58:28.931035 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aded15bb-391a-44da-b5dc-2e9ff016cfc8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:58:28.931499 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aded15bb-391a-44da-b5dc-2e9ff016cfc8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:58:28.931937 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aded15bb-391a-44da-b5dc-2e9ff016cfc8 admin admin] PATH_INFO: `/v3/users/03923cd0847f48d783361a92bdde030c` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:58:28.933446 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aded15bb-391a-44da-b5dc-2e9ff016cfc8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=03923cd0847f48d783361a92bdde030c)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:58:28.935976 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aded15bb-391a-44da-b5dc-2e9ff016cfc8 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:58:29.022166 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-aded15bb-391a-44da-b5dc-2e9ff016cfc8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '03923cd0847f48d783361a92bdde030c'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:58:29.055028 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-aded15bb-391a-44da-b5dc-2e9ff016cfc8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '03923cd0847f48d783361a92bdde030c'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:58:29.071464 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-aded15bb-391a-44da-b5dc-2e9ff016cfc8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '03923cd0847f48d783361a92bdde030c'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:58:29.091909 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-aded15bb-391a-44da-b5dc-2e9ff016cfc8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '03923cd0847f48d783361a92bdde030c'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:58:29.094201 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-aded15bb-391a-44da-b5dc-2e9ff016cfc8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '03923cd0847f48d783361a92bdde030c'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:58:29.125796 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1114/2240] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 00:58:28 2022] DELETE /identity/v3/users/03923cd0847f48d783361a92bdde030c => generated 0 bytes in 349 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:58:29.128142 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70bcb835-dcd6-4121-9f02-d04d3a6680de None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:58:29.128590 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70bcb835-dcd6-4121-9f02-d04d3a6680de None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:58:29.129083 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70bcb835-dcd6-4121-9f02-d04d3a6680de None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:58:29.171363 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-70bcb835-dcd6-4121-9f02-d04d3a6680de None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:58:29.176393 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-70bcb835-dcd6-4121-9f02-d04d3a6680de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:58:29.250915 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:58:29.274393 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1127/2241] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:58:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 28 00:58:29.287160 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aded15bb-391a-44da-b5dc-2e9ff016cfc8 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:58:29.290986 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-aded15bb-391a-44da-b5dc-2e9ff016cfc8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:58:29.388036 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aded15bb-391a-44da-b5dc-2e9ff016cfc8 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:58:29.394758 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-254abcdb-ff71-43b8-a070-27d167185701 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:58:29.395812 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-254abcdb-ff71-43b8-a070-27d167185701 service neutron] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:58:29.396306 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-254abcdb-ff71-43b8-a070-27d167185701 service neutron] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:58:29.396784 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-254abcdb-ff71-43b8-a070-27d167185701 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:58:29.413793 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-254abcdb-ff71-43b8-a070-27d167185701 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:58:29.416603 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-254abcdb-ff71-43b8-a070-27d167185701 service neutron] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:58:29.430603 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1115/2242] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:58:29 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 145 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:58:29.852715 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70bcb835-dcd6-4121-9f02-d04d3a6680de None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:58:29.854674 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-70bcb835-dcd6-4121-9f02-d04d3a6680de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:58:29.882634 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70bcb835-dcd6-4121-9f02-d04d3a6680de None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:58:29.892390 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ccae6ef-f909-4015-af7e-830428594548 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:58:29.893175 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ccae6ef-f909-4015-af7e-830428594548 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:58:29.893536 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ccae6ef-f909-4015-af7e-830428594548 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:58:29.893948 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ccae6ef-f909-4015-af7e-830428594548 admin admin] PATH_INFO: `/v3/projects/5feae4c5cde347b5a647c028f134f80d` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:58:29.895573 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ccae6ef-f909-4015-af7e-830428594548 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5feae4c5cde347b5a647c028f134f80d)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:58:29.898372 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ccae6ef-f909-4015-af7e-830428594548 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:58:29.956993 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-3ccae6ef-f909-4015-af7e-830428594548 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5feae4c5cde347b5a647c028f134f80d', 'request_id': 'req-3ccae6ef-f909-4015-af7e-830428594548'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:58:29.963654 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-3ccae6ef-f909-4015-af7e-830428594548 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5feae4c5cde347b5a647c028f134f80d', 'request_id': 'req-3ccae6ef-f909-4015-af7e-830428594548'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:58:29.971289 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-3ccae6ef-f909-4015-af7e-830428594548 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5feae4c5cde347b5a647c028f134f80d', 'request_id': 'req-3ccae6ef-f909-4015-af7e-830428594548'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:58:29.986880 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-3ccae6ef-f909-4015-af7e-830428594548 admin admin] The token cache is being invalidate because project 5feae4c5cde347b5a647c028f134f80d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81508) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 00:58:29.987228 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-3ccae6ef-f909-4015-af7e-830428594548 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:58:29.987959 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-3ccae6ef-f909-4015-af7e-830428594548 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 00:58:29.989632 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1128/2243] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 00:58:29 2022] DELETE /identity/v3/projects/5feae4c5cde347b5a647c028f134f80d => generated 0 bytes in 138 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 00:58:29.995096 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbf87590-eeea-4bb8-869a-57cdb9e6408c None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:58:29.995814 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbf87590-eeea-4bb8-869a-57cdb9e6408c None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:58:29.995814 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbf87590-eeea-4bb8-869a-57cdb9e6408c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:58:30.024000 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-dbf87590-eeea-4bb8-869a-57cdb9e6408c None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:58:30.026394 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-dbf87590-eeea-4bb8-869a-57cdb9e6408c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:58:30.065327 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:58:30.106829 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1116/2244] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:58:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:58:30.117600 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c4d229-dc1d-4150-b625-c515ebc59739 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:58:30.117991 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c4d229-dc1d-4150-b625-c515ebc59739 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:58:30.118332 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c4d229-dc1d-4150-b625-c515ebc59739 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:58:30.141882 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-81c4d229-dc1d-4150-b625-c515ebc59739 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:58:30.143244 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-81c4d229-dc1d-4150-b625-c515ebc59739 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:58:30.153315 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1129/2245] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:58:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:58:30.158111 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbf87590-eeea-4bb8-869a-57cdb9e6408c None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:58:30.169541 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbf87590-eeea-4bb8-869a-57cdb9e6408c None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:58:30.175445 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6588398-cc86-4e6d-8da4-fbc02b065926 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:58:30.177549 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6588398-cc86-4e6d-8da4-fbc02b065926 admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:58:30.178025 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6588398-cc86-4e6d-8da4-fbc02b065926 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:58:30.178470 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6588398-cc86-4e6d-8da4-fbc02b065926 admin admin] PATH_INFO: `/v3/domains` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:58:30.179642 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6588398-cc86-4e6d-8da4-fbc02b065926 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 00:58:30.180140 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6588398-cc86-4e6d-8da4-fbc02b065926 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:58:30.183220 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6588398-cc86-4e6d-8da4-fbc02b065926 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:58:30.204711 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1117/2246] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Aug 28 00:58:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 00:58:30.209624 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81c4d229-dc1d-4150-b625-c515ebc59739 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:58:30.222213 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81c4d229-dc1d-4150-b625-c515ebc59739 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:58:30.229109 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-995efc29-bdd1-4e41-8965-0aeb197992c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:58:30.230166 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-995efc29-bdd1-4e41-8965-0aeb197992c3 admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:58:30.230716 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-995efc29-bdd1-4e41-8965-0aeb197992c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:58:30.231168 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-995efc29-bdd1-4e41-8965-0aeb197992c3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:58:30.232171 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-995efc29-bdd1-4e41-8965-0aeb197992c3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:58:30.235246 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-995efc29-bdd1-4e41-8965-0aeb197992c3 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:58:30.289107 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1130/2247] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 00:58:30 2022] POST /identity/v3/projects => generated 383 bytes in 76 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 28 00:58:30.291057 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6588398-cc86-4e6d-8da4-fbc02b065926 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:58:30.307291 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6588398-cc86-4e6d-8da4-fbc02b065926 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:58:30.398675 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71429f1f-033d-44e0-9366-db9822e2c30e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:58:30.399757 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71429f1f-033d-44e0-9366-db9822e2c30e admin admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:58:30.400272 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71429f1f-033d-44e0-9366-db9822e2c30e admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:58:30.400786 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71429f1f-033d-44e0-9366-db9822e2c30e admin admin] PATH_INFO: `/v3/users` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:58:30.401903 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71429f1f-033d-44e0-9366-db9822e2c30e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:58:30.405507 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71429f1f-033d-44e0-9366-db9822e2c30e admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:58:30.438511 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1118/2248] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 00:58:30 2022] POST /identity/v3/users => generated 292 bytes in 148 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 00:58:30.443416 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-995efc29-bdd1-4e41-8965-0aeb197992c3 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:58:30.457890 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-995efc29-bdd1-4e41-8965-0aeb197992c3 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:58:30.464300 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-591b82b5-50e5-4837-a796-1117b4864e76 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:58:30.465185 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-591b82b5-50e5-4837-a796-1117b4864e76 admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:58:30.465624 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-591b82b5-50e5-4837-a796-1117b4864e76 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:58:30.466020 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-591b82b5-50e5-4837-a796-1117b4864e76 admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:58:30.466818 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-591b82b5-50e5-4837-a796-1117b4864e76 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:58:30.468642 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-591b82b5-50e5-4837-a796-1117b4864e76 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:58:30.483910 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1131/2249] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:58:30 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 00:58:30.489344 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71429f1f-033d-44e0-9366-db9822e2c30e admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:58:30.510894 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71429f1f-033d-44e0-9366-db9822e2c30e admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:58:30.519436 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7a7d565-6e28-4edc-bfab-aa958c80c1f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:58:30.520606 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7a7d565-6e28-4edc-bfab-aa958c80c1f1 admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:58:30.521130 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7a7d565-6e28-4edc-bfab-aa958c80c1f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:58:30.521625 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7a7d565-6e28-4edc-bfab-aa958c80c1f1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:58:30.522815 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7a7d565-6e28-4edc-bfab-aa958c80c1f1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:58:30.525821 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7a7d565-6e28-4edc-bfab-aa958c80c1f1 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:58:30.547075 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1119/2250] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 00:58:30 2022] GET /identity/v3/roles => generated 1420 bytes in 59 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 00:58:30.552593 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-591b82b5-50e5-4837-a796-1117b4864e76 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:58:30.565321 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-591b82b5-50e5-4837-a796-1117b4864e76 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:58:30.570812 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f3e786d-2120-4ea7-8bc4-73f8797fef93 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:58:30.571605 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f3e786d-2120-4ea7-8bc4-73f8797fef93 admin admin] REQUEST_METHOD: `PUT` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:58:30.572352 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f3e786d-2120-4ea7-8bc4-73f8797fef93 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:58:30.572879 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f3e786d-2120-4ea7-8bc4-73f8797fef93 admin admin] PATH_INFO: `/v3/projects/89dfee143ed147abb6f2339958412e7e/users/93ee4e8157364093bc9cdbe980bde4bb/roles/0487ef7e943e4701915cb8ce0f2899d7` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:58:30.592084 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f3e786d-2120-4ea7-8bc4-73f8797fef93 admin admin] RBAC: Authorizing `identity:create_grant(project_id=89dfee143ed147abb6f2339958412e7e, user_id=93ee4e8157364093bc9cdbe980bde4bb, role_id=0487ef7e943e4701915cb8ce0f2899d7)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:58:30.594978 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f3e786d-2120-4ea7-8bc4-73f8797fef93 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:58:30.613511 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1132/2251] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 00:58:30 2022] PUT /identity/v3/projects/89dfee143ed147abb6f2339958412e7e/users/93ee4e8157364093bc9cdbe980bde4bb/roles/0487ef7e943e4701915cb8ce0f2899d7 => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 28 00:58:30.619726 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd979e14-e88f-4fc9-9d7a-881cf0f1fc87 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:58:30.622608 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd979e14-e88f-4fc9-9d7a-881cf0f1fc87 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:58:30.623080 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd979e14-e88f-4fc9-9d7a-881cf0f1fc87 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:58:30.651452 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-cd979e14-e88f-4fc9-9d7a-881cf0f1fc87 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:58:30.654418 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-cd979e14-e88f-4fc9-9d7a-881cf0f1fc87 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:58:30.744809 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1120/2252] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:58:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:58:30.755719 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f3e786d-2120-4ea7-8bc4-73f8797fef93 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:58:30.757838 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-2f3e786d-2120-4ea7-8bc4-73f8797fef93 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:58:30.824403 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f3e786d-2120-4ea7-8bc4-73f8797fef93 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:58:30.828440 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-601dfe2f-067f-4534-93fc-a03aa6e7ec02 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:58:30.829218 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-601dfe2f-067f-4534-93fc-a03aa6e7ec02 service neutron] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:58:30.829557 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-601dfe2f-067f-4534-93fc-a03aa6e7ec02 service neutron] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:58:30.829935 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-601dfe2f-067f-4534-93fc-a03aa6e7ec02 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:58:30.841132 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-601dfe2f-067f-4534-93fc-a03aa6e7ec02 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:58:30.842601 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-601dfe2f-067f-4534-93fc-a03aa6e7ec02 service neutron] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:58:30.853249 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1133/2253] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:58:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:58:31.780727 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6134344-f8fd-478a-8f6a-3b79b34022bf None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:58:31.781359 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6134344-f8fd-478a-8f6a-3b79b34022bf None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:58:31.781773 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6134344-f8fd-478a-8f6a-3b79b34022bf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:58:31.886290 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-b6134344-f8fd-478a-8f6a-3b79b34022bf None None] MFA Rules not processed for user `93ee4e8157364093bc9cdbe980bde4bb`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:58:31.888827 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-b6134344-f8fd-478a-8f6a-3b79b34022bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:58:31.962355 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:58:32.050306 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1121/2254] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:58:31 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 272 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:58:32.073372 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-601dfe2f-067f-4534-93fc-a03aa6e7ec02 service neutron] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:58:32.081175 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-601dfe2f-067f-4534-93fc-a03aa6e7ec02 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:58:32.109281 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:58:32.229949 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-601dfe2f-067f-4534-93fc-a03aa6e7ec02 service neutron] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:58:32.233603 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b56b04b5-dc1a-40df-a3b1-88e4484c2b71 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:58:32.234236 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b56b04b5-dc1a-40df-a3b1-88e4484c2b71 service cinder] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:58:32.234533 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b56b04b5-dc1a-40df-a3b1-88e4484c2b71 service cinder] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:58:32.234805 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b56b04b5-dc1a-40df-a3b1-88e4484c2b71 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:58:32.244758 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b56b04b5-dc1a-40df-a3b1-88e4484c2b71 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:58:32.246424 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b56b04b5-dc1a-40df-a3b1-88e4484c2b71 service cinder] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:58:32.257863 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1134/2255] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:58:32 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 186 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:58:33.621964 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-412eee59-9f5c-4422-a996-992d6a8922f1 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:58:33.622609 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-412eee59-9f5c-4422-a996-992d6a8922f1 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:58:33.622943 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-412eee59-9f5c-4422-a996-992d6a8922f1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:58:33.656420 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-412eee59-9f5c-4422-a996-992d6a8922f1 None None] MFA Rules not processed for user `eede08f020e04282bb994b855e371433`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:58:33.658648 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-412eee59-9f5c-4422-a996-992d6a8922f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:58:33.792644 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1122/2256] 10.222.0.42 () {58 vars in 1009 bytes} [Sun Aug 28 00:58:33 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:58:33.798008 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b56b04b5-dc1a-40df-a3b1-88e4484c2b71 service cinder] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:58:33.801400 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-b56b04b5-dc1a-40df-a3b1-88e4484c2b71 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:58:33.922962 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b56b04b5-dc1a-40df-a3b1-88e4484c2b71 service cinder] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:58:33.928831 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e918aa98-6c22-4b85-8934-8ad57b071b6e service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac4c5584582f4f55a7085af4cbbabf9b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:58:33.929529 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e918aa98-6c22-4b85-8934-8ad57b071b6e service swift] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:58:33.929805 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e918aa98-6c22-4b85-8934-8ad57b071b6e service swift] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:58:33.930071 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e918aa98-6c22-4b85-8934-8ad57b071b6e service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:58:33.939974 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e918aa98-6c22-4b85-8934-8ad57b071b6e service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:58:33.941725 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e918aa98-6c22-4b85-8934-8ad57b071b6e service swift] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:58:33.953259 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1135/2257] 10.222.0.42 () {60 vars in 1384 bytes} [Sun Aug 28 00:58:33 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 157 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 28 00:58:34.984409 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb70d69c-9852-4d0b-807f-3e3721997310 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:58:34.985015 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb70d69c-9852-4d0b-807f-3e3721997310 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:58:34.985420 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb70d69c-9852-4d0b-807f-3e3721997310 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:58:35.056277 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-bb70d69c-9852-4d0b-807f-3e3721997310 None None] MFA Rules not processed for user `93ee4e8157364093bc9cdbe980bde4bb`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:58:35.059384 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-bb70d69c-9852-4d0b-807f-3e3721997310 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:58:35.080576 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1123/2258] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 00:58:34 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:58:43.948558 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=53 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:58:45.044724 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=41 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:58:56.095313 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-948aea97-9cdc-48e1-b82d-57f339d5b17b None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:58:56.095881 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-948aea97-9cdc-48e1-b82d-57f339d5b17b None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:58:56.095881 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-948aea97-9cdc-48e1-b82d-57f339d5b17b None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:58:56.097353 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1136/2259] 10.222.0.42 () {54 vars in 921 bytes} [Sun Aug 28 00:58:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:58:56.103836 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9c2b38f-ad7f-4e98-94e6-9089dd27a779 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:58:56.104548 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9c2b38f-ad7f-4e98-94e6-9089dd27a779 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:58:56.104548 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9c2b38f-ad7f-4e98-94e6-9089dd27a779 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:58:56.138549 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-c9c2b38f-ad7f-4e98-94e6-9089dd27a779 None None] MFA Rules not processed for user `168b07df50c24593ba73dbde5b037aed`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:58:56.139949 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-c9c2b38f-ad7f-4e98-94e6-9089dd27a779 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:58:56.233399 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1124/2260] 10.222.0.42 () {58 vars in 1018 bytes} [Sun Aug 28 00:58:56 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:58:56.251255 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-948aea97-9cdc-48e1-b82d-57f339d5b17b None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:58:56.254332 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-948aea97-9cdc-48e1-b82d-57f339d5b17b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:58:56.346434 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-948aea97-9cdc-48e1-b82d-57f339d5b17b None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:58:56.351633 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-357480e2-2356-4779-86d6-35b0689f0c75 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14176b54cf6640abb3b1c14e3824d4f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:58:56.352641 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-357480e2-2356-4779-86d6-35b0689f0c75 service glance] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:58:56.353017 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-357480e2-2356-4779-86d6-35b0689f0c75 service glance] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:58:56.353356 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-357480e2-2356-4779-86d6-35b0689f0c75 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:58:56.371155 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-357480e2-2356-4779-86d6-35b0689f0c75 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:58:56.374026 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-357480e2-2356-4779-86d6-35b0689f0c75 service glance] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:58:56.390535 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1137/2261] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:58:56 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 141 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:58:56.482488 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5759af49-96bf-49e5-b104-93c50d3bfca1 None None] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:58:56.482769 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5759af49-96bf-49e5-b104-93c50d3bfca1 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:58:56.483071 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5759af49-96bf-49e5-b104-93c50d3bfca1 None None] PATH_INFO: `/` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:58:56.484685 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1125/2262] 10.222.0.42 () {54 vars in 921 bytes} [Sun Aug 28 00:58:56 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Aug 28 00:58:56.490436 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae34e7d1-18ac-45d7-b6dd-caa10a5ed9ad None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:58:56.490805 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae34e7d1-18ac-45d7-b6dd-caa10a5ed9ad None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:58:56.491225 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae34e7d1-18ac-45d7-b6dd-caa10a5ed9ad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:58:56.517574 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-ae34e7d1-18ac-45d7-b6dd-caa10a5ed9ad None None] MFA Rules not processed for user `364a28c108e744728917ba2b0aa2347d`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:58:56.519088 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-ae34e7d1-18ac-45d7-b6dd-caa10a5ed9ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:58:56.543955 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1138/2263] 10.222.0.42 () {58 vars in 1018 bytes} [Sun Aug 28 00:58:56 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 55 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:58:56.560849 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5759af49-96bf-49e5-b104-93c50d3bfca1 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:58:56.563398 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-5759af49-96bf-49e5-b104-93c50d3bfca1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:58:56.584848 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5759af49-96bf-49e5-b104-93c50d3bfca1 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:58:56.604475 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f19dc4e-d009-41f1-b21d-95b44befd5be service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:58:56.605577 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f19dc4e-d009-41f1-b21d-95b44befd5be service neutron] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:58:56.606022 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f19dc4e-d009-41f1-b21d-95b44befd5be service neutron] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:58:56.606428 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f19dc4e-d009-41f1-b21d-95b44befd5be service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:58:56.619922 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f19dc4e-d009-41f1-b21d-95b44befd5be service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:58:56.622480 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f19dc4e-d009-41f1-b21d-95b44befd5be service neutron] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:58:56.638259 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1126/2264] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:58:56 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 78 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:58:56.756078 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feba7207-0ca0-456e-a05c-1c6a43181134 None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:58:56.756717 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feba7207-0ca0-456e-a05c-1c6a43181134 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:58:56.757144 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feba7207-0ca0-456e-a05c-1c6a43181134 None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:58:56.759735 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1139/2265] 10.222.0.42 () {54 vars in 921 bytes} [Sun Aug 28 00:58:56 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 00:58:56.769374 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b239356c-e786-46d0-aa86-304249bb7fe5 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:58:56.769654 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b239356c-e786-46d0-aa86-304249bb7fe5 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:58:56.769923 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b239356c-e786-46d0-aa86-304249bb7fe5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:58:56.800215 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-b239356c-e786-46d0-aa86-304249bb7fe5 None None] MFA Rules not processed for user `168b07df50c24593ba73dbde5b037aed`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:58:56.801823 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-b239356c-e786-46d0-aa86-304249bb7fe5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:58:56.817628 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1127/2266] 10.222.0.42 () {58 vars in 1018 bytes} [Sun Aug 28 00:58:56 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 49 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:58:56.831698 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feba7207-0ca0-456e-a05c-1c6a43181134 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:58:56.846791 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feba7207-0ca0-456e-a05c-1c6a43181134 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:58:56.851164 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfd5d638-f1ee-4bc8-8d3e-6b415cf85370 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:58:56.852113 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfd5d638-f1ee-4bc8-8d3e-6b415cf85370 service neutron] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:58:56.852540 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfd5d638-f1ee-4bc8-8d3e-6b415cf85370 service neutron] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:58:56.853128 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfd5d638-f1ee-4bc8-8d3e-6b415cf85370 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:58:56.866135 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfd5d638-f1ee-4bc8-8d3e-6b415cf85370 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:58:56.869054 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfd5d638-f1ee-4bc8-8d3e-6b415cf85370 service neutron] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:58:56.887384 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1140/2267] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:58:56 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 57 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 28 00:59:01.892950 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b3a7023-6556-489d-a9cb-f823f585ca64 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:59:01.893652 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b3a7023-6556-489d-a9cb-f823f585ca64 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:59:01.894129 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b3a7023-6556-489d-a9cb-f823f585ca64 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:59:01.927217 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-2b3a7023-6556-489d-a9cb-f823f585ca64 None None] MFA Rules not processed for user `eede08f020e04282bb994b855e371433`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:59:01.929388 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-2b3a7023-6556-489d-a9cb-f823f585ca64 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:59:01.942061 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1128/2268] 10.222.0.42 () {58 vars in 1009 bytes} [Sun Aug 28 00:59:01 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 51 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:59:01.972157 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfd5d638-f1ee-4bc8-8d3e-6b415cf85370 service neutron] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:59:01.987028 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfd5d638-f1ee-4bc8-8d3e-6b415cf85370 service neutron] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:59:01.995734 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8331e1a4-e5ac-4efe-8c85-b8d7aae3eb5c service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac4c5584582f4f55a7085af4cbbabf9b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:59:01.996759 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8331e1a4-e5ac-4efe-8c85-b8d7aae3eb5c service swift] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:59:01.997279 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8331e1a4-e5ac-4efe-8c85-b8d7aae3eb5c service swift] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:59:01.997794 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8331e1a4-e5ac-4efe-8c85-b8d7aae3eb5c service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:59:02.009216 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8331e1a4-e5ac-4efe-8c85-b8d7aae3eb5c service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:59:02.010948 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8331e1a4-e5ac-4efe-8c85-b8d7aae3eb5c service swift] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:59:02.020771 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1141/2269] 10.222.0.42 () {60 vars in 1384 bytes} [Sun Aug 28 00:59:01 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 49 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 28 00:59:11.919250 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=43 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:59:12.016872 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=41 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:59:40.853902 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cac9e8eb-f394-4375-8f9b-d9bf46fcdc9d None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:59:40.854768 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cac9e8eb-f394-4375-8f9b-d9bf46fcdc9d None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:59:40.854768 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cac9e8eb-f394-4375-8f9b-d9bf46fcdc9d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:59:40.881678 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-cac9e8eb-f394-4375-8f9b-d9bf46fcdc9d None None] MFA Rules not processed for user `eede08f020e04282bb994b855e371433`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 00:59:40.883030 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-cac9e8eb-f394-4375-8f9b-d9bf46fcdc9d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:59:40.892397 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1129/2270] 10.222.0.42 () {58 vars in 967 bytes} [Sun Aug 28 00:59:40 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 00:59:40.906124 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8331e1a4-e5ac-4efe-8c85-b8d7aae3eb5c service swift] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:59:40.918895 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8331e1a4-e5ac-4efe-8c85-b8d7aae3eb5c service swift] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:59:40.926002 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-239a2d91-1283-41ae-a535-4391d90813d1 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac4c5584582f4f55a7085af4cbbabf9b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:59:40.926685 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-239a2d91-1283-41ae-a535-4391d90813d1 service swift] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:59:40.927002 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-239a2d91-1283-41ae-a535-4391d90813d1 service swift] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:59:40.927312 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-239a2d91-1283-41ae-a535-4391d90813d1 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:59:40.938093 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-239a2d91-1283-41ae-a535-4391d90813d1 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:59:40.940994 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-239a2d91-1283-41ae-a535-4391d90813d1 service swift] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:59:40.966874 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1142/2271] 10.222.0.42 () {60 vars in 1384 bytes} [Sun Aug 28 00:59:40 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 61 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 28 00:59:42.635555 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cac9e8eb-f394-4375-8f9b-d9bf46fcdc9d None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:59:42.646949 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cac9e8eb-f394-4375-8f9b-d9bf46fcdc9d None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:59:42.651715 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36f50d01-6ece-40f7-8493-3e7b6b7623fa service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:59:42.652726 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36f50d01-6ece-40f7-8493-3e7b6b7623fa service neutron] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:59:42.653117 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36f50d01-6ece-40f7-8493-3e7b6b7623fa service neutron] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:59:42.653437 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36f50d01-6ece-40f7-8493-3e7b6b7623fa service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:59:42.655980 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-36f50d01-6ece-40f7-8493-3e7b6b7623fa service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:59:42.671364 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36f50d01-6ece-40f7-8493-3e7b6b7623fa service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:59:42.673039 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36f50d01-6ece-40f7-8493-3e7b6b7623fa service neutron] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:59:42.685683 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1130/2272] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:59:42 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 51 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:59:45.774988 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-239a2d91-1283-41ae-a535-4391d90813d1 service swift] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 00:59:45.784789 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-239a2d91-1283-41ae-a535-4391d90813d1 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:59:45.808461 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-239a2d91-1283-41ae-a535-4391d90813d1 service swift] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 00:59:45.815633 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5b2d018-2693-4d1f-b873-04fb91336378 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 00:59:45.816689 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5b2d018-2693-4d1f-b873-04fb91336378 service cinder] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 00:59:45.817220 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5b2d018-2693-4d1f-b873-04fb91336378 service cinder] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 00:59:45.817696 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5b2d018-2693-4d1f-b873-04fb91336378 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 00:59:45.821358 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-a5b2d018-2693-4d1f-b873-04fb91336378 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 00:59:45.928086 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5b2d018-2693-4d1f-b873-04fb91336378 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 00:59:45.930034 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5b2d018-2693-4d1f-b873-04fb91336378 service cinder] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 00:59:45.941845 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1143/2273] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 00:59:45 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 168 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 00:59:52.680076 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=9 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 00:59:55.936645 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=34 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:00:20.758447 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36f50d01-6ece-40f7-8493-3e7b6b7623fa service neutron] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:00:20.760630 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-36f50d01-6ece-40f7-8493-3e7b6b7623fa service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:00:20.777292 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36f50d01-6ece-40f7-8493-3e7b6b7623fa service neutron] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:00:20.784000 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee7b52d5-1ae0-40bd-a65e-87c28390e284 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '168b07df50c24593ba73dbde5b037aed', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:00:20.784770 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee7b52d5-1ae0-40bd-a65e-87c28390e284 service nova] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:00:20.785195 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee7b52d5-1ae0-40bd-a65e-87c28390e284 service nova] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:00:20.785606 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee7b52d5-1ae0-40bd-a65e-87c28390e284 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:00:20.787716 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-ee7b52d5-1ae0-40bd-a65e-87c28390e284 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:00:20.854526 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee7b52d5-1ae0-40bd-a65e-87c28390e284 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:00:20.856042 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee7b52d5-1ae0-40bd-a65e-87c28390e284 service nova] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:00:20.867013 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1131/2274] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:00:20 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 110 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:00:20.990673 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5b2d018-2693-4d1f-b873-04fb91336378 service cinder] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:00:21.004783 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5b2d018-2693-4d1f-b873-04fb91336378 service cinder] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:00:21.009213 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c133c1b-cdfb-4a22-8750-c1779e051fa4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:00:21.009835 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c133c1b-cdfb-4a22-8750-c1779e051fa4 service cinder] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:00:21.010220 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c133c1b-cdfb-4a22-8750-c1779e051fa4 service cinder] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:00:21.010372 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c133c1b-cdfb-4a22-8750-c1779e051fa4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:00:21.012242 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-9c133c1b-cdfb-4a22-8750-c1779e051fa4 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:00:21.026731 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c133c1b-cdfb-4a22-8750-c1779e051fa4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:00:21.028196 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c133c1b-cdfb-4a22-8750-c1779e051fa4 service cinder] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:00:21.040320 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1144/2275] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:00:20 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 51 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:00:30.861733 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=28 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:00:31.034990 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=6 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:00:33.958589 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee7b52d5-1ae0-40bd-a65e-87c28390e284 service nova] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:00:33.960615 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-ee7b52d5-1ae0-40bd-a65e-87c28390e284 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:00:33.976526 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee7b52d5-1ae0-40bd-a65e-87c28390e284 service nova] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:00:33.983586 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a63e07f-abf5-41a7-b7d8-3c28cde70ed9 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '168b07df50c24593ba73dbde5b037aed', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:00:33.984283 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a63e07f-abf5-41a7-b7d8-3c28cde70ed9 service nova] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:00:33.984562 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a63e07f-abf5-41a7-b7d8-3c28cde70ed9 service nova] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:00:33.984862 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a63e07f-abf5-41a7-b7d8-3c28cde70ed9 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:00:33.987050 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-0a63e07f-abf5-41a7-b7d8-3c28cde70ed9 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:00:34.065600 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a63e07f-abf5-41a7-b7d8-3c28cde70ed9 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:00:34.067720 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a63e07f-abf5-41a7-b7d8-3c28cde70ed9 service nova] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:00:34.080889 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1132/2276] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:00:33 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 124 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:00:39.950993 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c133c1b-cdfb-4a22-8750-c1779e051fa4 service cinder] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:00:39.969853 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c133c1b-cdfb-4a22-8750-c1779e051fa4 service cinder] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:00:39.978535 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65728e6f-34a6-47fe-b268-ee9b4b4399c0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:00:39.979650 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65728e6f-34a6-47fe-b268-ee9b4b4399c0 service neutron] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:00:39.980099 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65728e6f-34a6-47fe-b268-ee9b4b4399c0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:00:39.980796 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65728e6f-34a6-47fe-b268-ee9b4b4399c0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:00:39.984482 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-65728e6f-34a6-47fe-b268-ee9b4b4399c0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:00:40.006146 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65728e6f-34a6-47fe-b268-ee9b4b4399c0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:00:40.008834 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65728e6f-34a6-47fe-b268-ee9b4b4399c0 service neutron] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:00:40.023719 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1145/2277] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:00:39 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 73 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:00:43.384461 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a63e07f-abf5-41a7-b7d8-3c28cde70ed9 service nova] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:00:43.409089 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a63e07f-abf5-41a7-b7d8-3c28cde70ed9 service nova] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:00:43.420471 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b84c21d8-6233-4bd9-ae11-07aa27222134 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '168b07df50c24593ba73dbde5b037aed', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:00:43.421521 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b84c21d8-6233-4bd9-ae11-07aa27222134 service nova] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:00:43.422014 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b84c21d8-6233-4bd9-ae11-07aa27222134 service nova] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:00:43.423019 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b84c21d8-6233-4bd9-ae11-07aa27222134 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:00:43.425901 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-b84c21d8-6233-4bd9-ae11-07aa27222134 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:00:43.450380 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b84c21d8-6233-4bd9-ae11-07aa27222134 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:00:43.453545 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b84c21d8-6233-4bd9-ae11-07aa27222134 service nova] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:00:43.475582 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1133/2278] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:00:43 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:00:44.665615 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65728e6f-34a6-47fe-b268-ee9b4b4399c0 service neutron] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:00:44.690387 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65728e6f-34a6-47fe-b268-ee9b4b4399c0 service neutron] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:00:44.699617 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f06eed3a-6081-43eb-a2c9-df18a9eb8f19 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '168b07df50c24593ba73dbde5b037aed', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:00:44.700890 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f06eed3a-6081-43eb-a2c9-df18a9eb8f19 service nova] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:00:44.702792 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f06eed3a-6081-43eb-a2c9-df18a9eb8f19 service nova] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:00:44.705046 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f06eed3a-6081-43eb-a2c9-df18a9eb8f19 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:00:44.709642 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-f06eed3a-6081-43eb-a2c9-df18a9eb8f19 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:00:44.748168 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f06eed3a-6081-43eb-a2c9-df18a9eb8f19 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:00:44.750999 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f06eed3a-6081-43eb-a2c9-df18a9eb8f19 service nova] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:00:44.787629 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1146/2279] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:00:44 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 123 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:00:50.102792 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b84c21d8-6233-4bd9-ae11-07aa27222134 service nova] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:00:50.118276 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b84c21d8-6233-4bd9-ae11-07aa27222134 service nova] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:00:50.125608 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24ac18fc-cc79-4659-adbf-9cc1296a188b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '168b07df50c24593ba73dbde5b037aed', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:00:50.126484 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24ac18fc-cc79-4659-adbf-9cc1296a188b service nova] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:00:50.126484 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24ac18fc-cc79-4659-adbf-9cc1296a188b service nova] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:00:50.126770 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24ac18fc-cc79-4659-adbf-9cc1296a188b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:00:50.128873 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-24ac18fc-cc79-4659-adbf-9cc1296a188b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:00:50.198436 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24ac18fc-cc79-4659-adbf-9cc1296a188b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:00:50.200114 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24ac18fc-cc79-4659-adbf-9cc1296a188b service nova] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:00:50.211142 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1134/2280] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:00:50 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 110 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:00:54.772426 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=12 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:01:00.206286 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=62 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:01:04.235173 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65f70294-9168-4a11-adda-1cda8e4cd210 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:01:04.235792 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65f70294-9168-4a11-adda-1cda8e4cd210 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:01:04.236133 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65f70294-9168-4a11-adda-1cda8e4cd210 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:01:04.263710 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-65f70294-9168-4a11-adda-1cda8e4cd210 None None] MFA Rules not processed for user `1ab2fcb3759045c7a076e5c8222a421d`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:01:04.265191 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-65f70294-9168-4a11-adda-1cda8e4cd210 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:01:04.274618 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1147/2281] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:01:04 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:01:14.255706 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=3 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:02:20.882539 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24ac18fc-cc79-4659-adbf-9cc1296a188b service nova] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:02:20.901530 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24ac18fc-cc79-4659-adbf-9cc1296a188b service nova] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:02:20.911028 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c622604b-2452-46e4-95df-d6bbb76d701a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '168b07df50c24593ba73dbde5b037aed', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:02:20.911936 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c622604b-2452-46e4-95df-d6bbb76d701a service nova] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:02:20.912410 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c622604b-2452-46e4-95df-d6bbb76d701a service nova] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:02:20.912862 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c622604b-2452-46e4-95df-d6bbb76d701a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:02:20.915568 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-c622604b-2452-46e4-95df-d6bbb76d701a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:02:20.994817 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c622604b-2452-46e4-95df-d6bbb76d701a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:02:20.997402 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c622604b-2452-46e4-95df-d6bbb76d701a service nova] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:02:21.015638 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1135/2282] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:02:20 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 134 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:02:31.006964 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=28 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:03:56.561773 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9545ea61-8d3b-461e-b9d9-4dc069d656a8 None None] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:03:56.562526 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9545ea61-8d3b-461e-b9d9-4dc069d656a8 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:03:56.562526 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9545ea61-8d3b-461e-b9d9-4dc069d656a8 None None] PATH_INFO: `/` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:03:56.564142 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1148/2283] 10.222.0.42 () {54 vars in 961 bytes} [Sun Aug 28 01:03:56 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Aug 28 01:03:56.573549 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e636ddbb-65ee-40cb-9a74-543990ed2b7e None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:03:56.574717 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e636ddbb-65ee-40cb-9a74-543990ed2b7e None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:03:56.574717 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e636ddbb-65ee-40cb-9a74-543990ed2b7e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:03:56.681871 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-e636ddbb-65ee-40cb-9a74-543990ed2b7e None None] MFA Rules not processed for user `c2a3834fa6894dada3291e4781bbd553`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:03:56.683148 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-e636ddbb-65ee-40cb-9a74-543990ed2b7e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:03:56.816524 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1136/2284] 10.222.0.42 () {58 vars in 1058 bytes} [Sun Aug 28 01:03:56 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 245 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:03:56.825154 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9545ea61-8d3b-461e-b9d9-4dc069d656a8 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:03:56.839693 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9545ea61-8d3b-461e-b9d9-4dc069d656a8 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:03:56.845276 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2dc9652-d8c2-4cb1-8374-37df252d63bc service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2a3834fa6894dada3291e4781bbd553', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:03:56.846019 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2dc9652-d8c2-4cb1-8374-37df252d63bc service placement] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:03:56.846351 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2dc9652-d8c2-4cb1-8374-37df252d63bc service placement] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:03:56.846663 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2dc9652-d8c2-4cb1-8374-37df252d63bc service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:03:56.848898 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-c2dc9652-d8c2-4cb1-8374-37df252d63bc service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:03:56.862796 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2dc9652-d8c2-4cb1-8374-37df252d63bc service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:03:56.864477 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2dc9652-d8c2-4cb1-8374-37df252d63bc service placement] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:03:56.876429 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1149/2285] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:03:56 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 52 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:04:06.813552 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=74 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:04:06.871017 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=6 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:04:42.234235 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e636ddbb-65ee-40cb-9a74-543990ed2b7e None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:04:42.267217 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e636ddbb-65ee-40cb-9a74-543990ed2b7e None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:04:42.272361 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-794df800-2482-4566-864b-94b956b7c587 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:04:42.273080 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-794df800-2482-4566-864b-94b956b7c587 service cinder] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:04:42.273365 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-794df800-2482-4566-864b-94b956b7c587 service cinder] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:04:42.273663 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-794df800-2482-4566-864b-94b956b7c587 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:04:42.284467 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-794df800-2482-4566-864b-94b956b7c587 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:04:42.286009 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-794df800-2482-4566-864b-94b956b7c587 service cinder] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:04:42.297899 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1137/2286] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:04:42 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 65 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:04:52.292428 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=10 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:05:13.853361 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2dc9652-d8c2-4cb1-8374-37df252d63bc service placement] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:05:13.869816 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2dc9652-d8c2-4cb1-8374-37df252d63bc service placement] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:05:13.874673 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-229da6ca-1b0c-459c-b273-45c49bf37cd2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:05:13.875307 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-229da6ca-1b0c-459c-b273-45c49bf37cd2 service cinder] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:05:13.875587 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-229da6ca-1b0c-459c-b273-45c49bf37cd2 service cinder] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:05:13.875861 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-229da6ca-1b0c-459c-b273-45c49bf37cd2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:05:13.897013 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-229da6ca-1b0c-459c-b273-45c49bf37cd2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:05:13.898507 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-229da6ca-1b0c-459c-b273-45c49bf37cd2 service cinder] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:05:13.923219 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1150/2287] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:05:13 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 70 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:05:13.931125 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-794df800-2482-4566-864b-94b956b7c587 service cinder] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:05:13.941136 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-794df800-2482-4566-864b-94b956b7c587 service cinder] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:05:13.945372 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30b62ab5-810e-4efd-92a5-6f577b803673 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:05:13.946019 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30b62ab5-810e-4efd-92a5-6f577b803673 service cinder] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:05:13.946296 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30b62ab5-810e-4efd-92a5-6f577b803673 service cinder] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:05:13.946561 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30b62ab5-810e-4efd-92a5-6f577b803673 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:05:13.956647 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30b62ab5-810e-4efd-92a5-6f577b803673 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:05:13.958164 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30b62ab5-810e-4efd-92a5-6f577b803673 service cinder] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:05:13.989496 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1138/2288] 10.222.0.42 () {60 vars in 1396 bytes} [Sun Aug 28 01:05:13 2022] GET /identity/v3/auth/tokens?allow_expired=1 => generated 3268 bytes in 59 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 28 01:05:23.920146 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=13 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:05:23.986418 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=13 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:05:34.393834 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-229da6ca-1b0c-459c-b273-45c49bf37cd2 service cinder] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:05:34.406190 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-229da6ca-1b0c-459c-b273-45c49bf37cd2 service cinder] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:05:34.411926 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdf0fec1-18b3-4ddb-87f9-11769c23402f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '168b07df50c24593ba73dbde5b037aed', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:05:34.412551 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf0fec1-18b3-4ddb-87f9-11769c23402f service nova] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:05:34.412824 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf0fec1-18b3-4ddb-87f9-11769c23402f service nova] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:05:34.413124 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf0fec1-18b3-4ddb-87f9-11769c23402f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:05:34.422188 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdf0fec1-18b3-4ddb-87f9-11769c23402f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:05:34.423874 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdf0fec1-18b3-4ddb-87f9-11769c23402f service nova] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:05:34.453834 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1151/2289] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:05:34 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 61 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:05:44.450966 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=13 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:05:56.254701 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30b62ab5-810e-4efd-92a5-6f577b803673 service cinder] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:05:56.276450 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30b62ab5-810e-4efd-92a5-6f577b803673 service cinder] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:05:56.281049 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-283f6dbe-156a-44f4-9110-36c6c2f27e01 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:05:56.281670 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-283f6dbe-156a-44f4-9110-36c6c2f27e01 service neutron] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:05:56.281944 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-283f6dbe-156a-44f4-9110-36c6c2f27e01 service neutron] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:05:56.282215 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-283f6dbe-156a-44f4-9110-36c6c2f27e01 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:05:56.290946 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-283f6dbe-156a-44f4-9110-36c6c2f27e01 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:05:56.292428 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-283f6dbe-156a-44f4-9110-36c6c2f27e01 service neutron] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:05:56.301129 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1139/2290] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:05:56 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 47 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:06:06.297480 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=10 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:06:42.525605 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdf0fec1-18b3-4ddb-87f9-11769c23402f service nova] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:06:42.536549 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdf0fec1-18b3-4ddb-87f9-11769c23402f service nova] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:06:42.541220 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f14ed8c0-dd75-4475-a7f0-9ef42cdf5095 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:06:42.541920 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f14ed8c0-dd75-4475-a7f0-9ef42cdf5095 service cinder] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:06:42.542216 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f14ed8c0-dd75-4475-a7f0-9ef42cdf5095 service cinder] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:06:42.542470 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f14ed8c0-dd75-4475-a7f0-9ef42cdf5095 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:06:42.554923 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f14ed8c0-dd75-4475-a7f0-9ef42cdf5095 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:06:42.556426 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f14ed8c0-dd75-4475-a7f0-9ef42cdf5095 service cinder] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:06:42.569882 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1152/2291] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:06:42 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:06:42.617609 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-283f6dbe-156a-44f4-9110-36c6c2f27e01 service neutron] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:06:42.629187 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-283f6dbe-156a-44f4-9110-36c6c2f27e01 service neutron] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:06:42.632859 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf48458c-ba40-4775-a34b-b67b78bbe701 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:06:42.633535 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf48458c-ba40-4775-a34b-b67b78bbe701 service cinder] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:06:42.633810 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf48458c-ba40-4775-a34b-b67b78bbe701 service cinder] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:06:42.634081 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf48458c-ba40-4775-a34b-b67b78bbe701 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:06:42.643418 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf48458c-ba40-4775-a34b-b67b78bbe701 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:06:42.644856 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf48458c-ba40-4775-a34b-b67b78bbe701 service cinder] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:06:42.679996 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1140/2292] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:06:42 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 64 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:06:42.727943 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f14ed8c0-dd75-4475-a7f0-9ef42cdf5095 service cinder] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:06:42.740272 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f14ed8c0-dd75-4475-a7f0-9ef42cdf5095 service cinder] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:06:42.748757 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a33c20bf-0845-4da8-a2ab-5cb5bb7ea5d6 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '168b07df50c24593ba73dbde5b037aed', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:06:42.749411 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a33c20bf-0845-4da8-a2ab-5cb5bb7ea5d6 service nova] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:06:42.749708 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a33c20bf-0845-4da8-a2ab-5cb5bb7ea5d6 service nova] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:06:42.749976 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a33c20bf-0845-4da8-a2ab-5cb5bb7ea5d6 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:06:42.759232 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a33c20bf-0845-4da8-a2ab-5cb5bb7ea5d6 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:06:42.760870 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a33c20bf-0845-4da8-a2ab-5cb5bb7ea5d6 service nova] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:06:42.791697 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1153/2293] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:06:42 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 64 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:06:52.675628 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=13 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:06:52.788524 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=19 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:08:43.026251 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf48458c-ba40-4775-a34b-b67b78bbe701 service cinder] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:08:43.048492 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf48458c-ba40-4775-a34b-b67b78bbe701 service cinder] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:08:43.058248 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c4fe85a-60ba-498d-b620-5b4ef3902afe service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '168b07df50c24593ba73dbde5b037aed', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:08:43.059296 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c4fe85a-60ba-498d-b620-5b4ef3902afe service nova] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:08:43.059754 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c4fe85a-60ba-498d-b620-5b4ef3902afe service nova] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:08:43.060223 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c4fe85a-60ba-498d-b620-5b4ef3902afe service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:08:43.075656 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c4fe85a-60ba-498d-b620-5b4ef3902afe service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:08:43.077437 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c4fe85a-60ba-498d-b620-5b4ef3902afe service nova] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:08:43.115494 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1141/2294] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:08:43 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:08:53.108009 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=13 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:09:02.521851 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a33c20bf-0845-4da8-a2ab-5cb5bb7ea5d6 service nova] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:09:02.535030 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a33c20bf-0845-4da8-a2ab-5cb5bb7ea5d6 service nova] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:09:02.541465 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0d0af73-274e-4868-8ff0-57d168b67c4d service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2a3834fa6894dada3291e4781bbd553', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:09:02.542115 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d0af73-274e-4868-8ff0-57d168b67c4d service placement] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:09:02.542391 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d0af73-274e-4868-8ff0-57d168b67c4d service placement] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:09:02.542661 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d0af73-274e-4868-8ff0-57d168b67c4d service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:09:02.554613 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0d0af73-274e-4868-8ff0-57d168b67c4d service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:09:02.557548 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0d0af73-274e-4868-8ff0-57d168b67c4d service placement] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:09:02.576739 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1154/2295] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:09:02 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 55 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:09:12.567596 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=6 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:10:43.039154 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c4fe85a-60ba-498d-b620-5b4ef3902afe service nova] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:10:43.040805 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-1c4fe85a-60ba-498d-b620-5b4ef3902afe service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:10:43.118654 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c4fe85a-60ba-498d-b620-5b4ef3902afe service nova] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:10:43.123008 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2638ec2b-3254-47d8-8964-e51e5b38d3e5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:10:43.123623 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2638ec2b-3254-47d8-8964-e51e5b38d3e5 service cinder] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:10:43.123949 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2638ec2b-3254-47d8-8964-e51e5b38d3e5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:10:43.124221 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2638ec2b-3254-47d8-8964-e51e5b38d3e5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:10:43.126186 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-2638ec2b-3254-47d8-8964-e51e5b38d3e5 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:10:43.350852 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2638ec2b-3254-47d8-8964-e51e5b38d3e5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:10:43.352185 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2638ec2b-3254-47d8-8964-e51e5b38d3e5 service cinder] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:10:43.365149 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1142/2296] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:10:43 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 327 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:10:43.374030 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0d0af73-274e-4868-8ff0-57d168b67c4d service placement] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:10:43.385685 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0d0af73-274e-4868-8ff0-57d168b67c4d service placement] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:10:43.390216 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f890a6e-b56a-4ee5-81e0-5c3b8f008c3a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:10:43.390858 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f890a6e-b56a-4ee5-81e0-5c3b8f008c3a service cinder] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:10:43.391167 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f890a6e-b56a-4ee5-81e0-5c3b8f008c3a service cinder] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:10:43.391392 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f890a6e-b56a-4ee5-81e0-5c3b8f008c3a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:10:43.393847 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-2f890a6e-b56a-4ee5-81e0-5c3b8f008c3a service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:10:43.479708 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f890a6e-b56a-4ee5-81e0-5c3b8f008c3a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:10:43.481206 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f890a6e-b56a-4ee5-81e0-5c3b8f008c3a service cinder] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:10:43.500784 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1155/2297] 10.222.0.42 () {60 vars in 1396 bytes} [Sun Aug 28 01:10:43 2022] GET /identity/v3/auth/tokens?allow_expired=1 => generated 3260 bytes in 127 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:10:43.553177 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2638ec2b-3254-47d8-8964-e51e5b38d3e5 service cinder] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:10:43.562576 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2638ec2b-3254-47d8-8964-e51e5b38d3e5 service cinder] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:10:43.566885 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5bf2156-3bbf-4e02-a60c-5c4a375f34da service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:10:43.567527 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5bf2156-3bbf-4e02-a60c-5c4a375f34da service cinder] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:10:43.567779 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5bf2156-3bbf-4e02-a60c-5c4a375f34da service cinder] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:10:43.568048 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5bf2156-3bbf-4e02-a60c-5c4a375f34da service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:10:43.570430 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-b5bf2156-3bbf-4e02-a60c-5c4a375f34da service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:10:43.647087 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5bf2156-3bbf-4e02-a60c-5c4a375f34da service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:10:43.648574 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5bf2156-3bbf-4e02-a60c-5c4a375f34da service cinder] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:10:43.659826 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1143/2298] 10.222.0.42 () {60 vars in 1396 bytes} [Sun Aug 28 01:10:43 2022] GET /identity/v3/auth/tokens?allow_expired=1 => generated 3268 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:10:43.819932 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f890a6e-b56a-4ee5-81e0-5c3b8f008c3a service cinder] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:10:43.821698 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-2f890a6e-b56a-4ee5-81e0-5c3b8f008c3a service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:10:43.842529 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f890a6e-b56a-4ee5-81e0-5c3b8f008c3a service cinder] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:10:43.853436 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5bf2156-3bbf-4e02-a60c-5c4a375f34da service cinder] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:10:43.853899 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abef23b6-b716-4216-bf5c-72543b845449 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '168b07df50c24593ba73dbde5b037aed', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:10:43.854531 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abef23b6-b716-4216-bf5c-72543b845449 service nova] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:10:43.854820 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abef23b6-b716-4216-bf5c-72543b845449 service nova] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:10:43.855127 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abef23b6-b716-4216-bf5c-72543b845449 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:10:43.857337 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-abef23b6-b716-4216-bf5c-72543b845449 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:10:43.865349 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5bf2156-3bbf-4e02-a60c-5c4a375f34da service cinder] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:10:43.872321 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a09d687c-899b-4bbe-85a2-d607ac9f9efe service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:10:43.873094 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a09d687c-899b-4bbe-85a2-d607ac9f9efe service cinder] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:10:43.873380 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a09d687c-899b-4bbe-85a2-d607ac9f9efe service cinder] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:10:43.873654 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a09d687c-899b-4bbe-85a2-d607ac9f9efe service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:10:43.875755 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-a09d687c-899b-4bbe-85a2-d607ac9f9efe service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:10:43.880871 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:10:43.945335 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abef23b6-b716-4216-bf5c-72543b845449 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:10:43.946869 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abef23b6-b716-4216-bf5c-72543b845449 service nova] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:10:43.957531 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1156/2299] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:10:43 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 139 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:10:43.963160 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a09d687c-899b-4bbe-85a2-d607ac9f9efe service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:10:43.964674 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a09d687c-899b-4bbe-85a2-d607ac9f9efe service cinder] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:10:43.978846 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1144/2300] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:10:43 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 126 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 28 01:10:53.952421 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=74 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:10:53.971238 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=30 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:11:04.161796 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abef23b6-b716-4216-bf5c-72543b845449 service nova] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:11:04.163704 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-abef23b6-b716-4216-bf5c-72543b845449 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:11:04.240284 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abef23b6-b716-4216-bf5c-72543b845449 service nova] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:11:04.244247 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f62f391-69e8-4fa9-ac9e-d0e92af62bb1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:11:04.244859 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f62f391-69e8-4fa9-ac9e-d0e92af62bb1 service neutron] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:11:04.245164 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f62f391-69e8-4fa9-ac9e-d0e92af62bb1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:11:04.245428 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f62f391-69e8-4fa9-ac9e-d0e92af62bb1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:11:04.247432 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-4f62f391-69e8-4fa9-ac9e-d0e92af62bb1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:11:04.257859 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f62f391-69e8-4fa9-ac9e-d0e92af62bb1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:11:04.259288 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f62f391-69e8-4fa9-ac9e-d0e92af62bb1 service neutron] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:11:04.267745 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1157/2301] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:11:04 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:11:14.264052 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=31 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:11:15.455737 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-754c2ed2-fe68-4ede-ae30-2e376c3e0fb8 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:11:15.455737 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-754c2ed2-fe68-4ede-ae30-2e376c3e0fb8 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:11:15.455737 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-754c2ed2-fe68-4ede-ae30-2e376c3e0fb8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:11:15.518902 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-754c2ed2-fe68-4ede-ae30-2e376c3e0fb8 None None] MFA Rules not processed for user `cac1ab72dcee40b19bcff4dd5a75d82d`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:11:15.521451 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-754c2ed2-fe68-4ede-ae30-2e376c3e0fb8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:11:15.531516 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1145/2302] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:11:15 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:11:15.919165 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f62f391-69e8-4fa9-ac9e-d0e92af62bb1 service neutron] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:11:15.928466 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f62f391-69e8-4fa9-ac9e-d0e92af62bb1 service neutron] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:11:15.932560 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9ffc2aa-2a81-4bd6-b3e7-4e082a66cd75 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:11:15.933160 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9ffc2aa-2a81-4bd6-b3e7-4e082a66cd75 service neutron] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:11:15.933454 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9ffc2aa-2a81-4bd6-b3e7-4e082a66cd75 service neutron] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:11:15.933721 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9ffc2aa-2a81-4bd6-b3e7-4e082a66cd75 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:11:15.935499 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-a9ffc2aa-2a81-4bd6-b3e7-4e082a66cd75 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:11:15.945787 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9ffc2aa-2a81-4bd6-b3e7-4e082a66cd75 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:11:15.947219 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9ffc2aa-2a81-4bd6-b3e7-4e082a66cd75 service neutron] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:11:15.955474 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1158/2303] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:11:15 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:11:25.515165 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=11 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:11:25.951876 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=6 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:11:41.222749 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-754c2ed2-fe68-4ede-ae30-2e376c3e0fb8 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:11:41.224494 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-754c2ed2-fe68-4ede-ae30-2e376c3e0fb8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:11:41.241920 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-754c2ed2-fe68-4ede-ae30-2e376c3e0fb8 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:11:41.248162 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2e79092-befa-41df-be31-0ae78269e115 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '168b07df50c24593ba73dbde5b037aed', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:11:41.248808 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2e79092-befa-41df-be31-0ae78269e115 service nova] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:11:41.249128 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2e79092-befa-41df-be31-0ae78269e115 service nova] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:11:41.249401 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2e79092-befa-41df-be31-0ae78269e115 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:11:41.251559 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-e2e79092-befa-41df-be31-0ae78269e115 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:11:41.323546 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2e79092-befa-41df-be31-0ae78269e115 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:11:41.325256 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2e79092-befa-41df-be31-0ae78269e115 service nova] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:11:41.334565 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1146/2304] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:11:41 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:11:42.202402 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9ffc2aa-2a81-4bd6-b3e7-4e082a66cd75 service neutron] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:11:42.204260 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-a9ffc2aa-2a81-4bd6-b3e7-4e082a66cd75 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:11:42.216296 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9ffc2aa-2a81-4bd6-b3e7-4e082a66cd75 service neutron] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:11:42.220276 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3373bd23-0ff1-4bfe-99f9-40713652e698 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:11:42.220895 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3373bd23-0ff1-4bfe-99f9-40713652e698 service cinder] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:11:42.221189 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3373bd23-0ff1-4bfe-99f9-40713652e698 service cinder] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:11:42.221472 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3373bd23-0ff1-4bfe-99f9-40713652e698 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:11:42.223478 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-3373bd23-0ff1-4bfe-99f9-40713652e698 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:11:42.307450 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3373bd23-0ff1-4bfe-99f9-40713652e698 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:11:42.308989 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3373bd23-0ff1-4bfe-99f9-40713652e698 service cinder] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:11:42.318970 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1159/2305] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:11:42 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 117 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:11:51.330412 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=33 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:11:52.314543 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=33 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:13:42.951572 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2e79092-befa-41df-be31-0ae78269e115 service nova] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:13:42.961690 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2e79092-befa-41df-be31-0ae78269e115 service nova] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:13:42.966062 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-675a3126-eb2e-40c1-a091-7fdaf1735ca9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:13:42.966669 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-675a3126-eb2e-40c1-a091-7fdaf1735ca9 service cinder] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:13:42.966946 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-675a3126-eb2e-40c1-a091-7fdaf1735ca9 service cinder] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:13:42.967211 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-675a3126-eb2e-40c1-a091-7fdaf1735ca9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:13:42.969218 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-675a3126-eb2e-40c1-a091-7fdaf1735ca9 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:13:43.029586 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-675a3126-eb2e-40c1-a091-7fdaf1735ca9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:13:43.031020 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-675a3126-eb2e-40c1-a091-7fdaf1735ca9 service cinder] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:13:43.039891 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1147/2306] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:13:42 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:13:53.035734 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=28 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:14:09.510367 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3373bd23-0ff1-4bfe-99f9-40713652e698 service cinder] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:14:09.512794 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-3373bd23-0ff1-4bfe-99f9-40713652e698 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:14:09.679440 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3373bd23-0ff1-4bfe-99f9-40713652e698 service cinder] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:14:09.685888 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-815e0c94-a96a-419b-bed1-7ebbdc249ece service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2a3834fa6894dada3291e4781bbd553', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:14:09.686513 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-815e0c94-a96a-419b-bed1-7ebbdc249ece service placement] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:14:09.686793 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-815e0c94-a96a-419b-bed1-7ebbdc249ece service placement] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:14:09.687069 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-815e0c94-a96a-419b-bed1-7ebbdc249ece service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:14:09.689153 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-815e0c94-a96a-419b-bed1-7ebbdc249ece service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:14:09.705328 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-815e0c94-a96a-419b-bed1-7ebbdc249ece service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:14:09.707783 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-815e0c94-a96a-419b-bed1-7ebbdc249ece service placement] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:14:09.717736 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1160/2307] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:14:09 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 208 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 28 01:14:19.713644 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=63 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:15:43.666306 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-675a3126-eb2e-40c1-a091-7fdaf1735ca9 service cinder] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:15:43.730939 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-675a3126-eb2e-40c1-a091-7fdaf1735ca9 service cinder] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:15:43.736901 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc4b3be3-6e56-46c3-b5b7-1f44fa48522f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '168b07df50c24593ba73dbde5b037aed', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:15:43.737574 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc4b3be3-6e56-46c3-b5b7-1f44fa48522f service nova] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:15:43.737859 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc4b3be3-6e56-46c3-b5b7-1f44fa48522f service nova] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:15:43.738129 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc4b3be3-6e56-46c3-b5b7-1f44fa48522f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:15:43.746794 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc4b3be3-6e56-46c3-b5b7-1f44fa48522f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:15:43.748407 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc4b3be3-6e56-46c3-b5b7-1f44fa48522f service nova] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:15:43.778344 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1148/2308] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:15:43 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:15:43.913741 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-815e0c94-a96a-419b-bed1-7ebbdc249ece service placement] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:15:43.933776 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-815e0c94-a96a-419b-bed1-7ebbdc249ece service placement] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:15:43.937470 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72ccd051-4973-4c99-bd2b-707b277b4bee service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:15:43.938083 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72ccd051-4973-4c99-bd2b-707b277b4bee service cinder] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:15:43.938359 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72ccd051-4973-4c99-bd2b-707b277b4bee service cinder] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:15:43.938631 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72ccd051-4973-4c99-bd2b-707b277b4bee service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:15:43.948331 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72ccd051-4973-4c99-bd2b-707b277b4bee service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:15:43.949830 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72ccd051-4973-4c99-bd2b-707b277b4bee service cinder] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:15:43.959965 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1161/2309] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:15:43 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 47 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:15:44.160097 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc4b3be3-6e56-46c3-b5b7-1f44fa48522f service nova] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:15:44.170415 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc4b3be3-6e56-46c3-b5b7-1f44fa48522f service nova] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:15:44.174828 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80b564ce-17a9-4594-b542-7f7513d0b000 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:15:44.175465 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80b564ce-17a9-4594-b542-7f7513d0b000 service cinder] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:15:44.175743 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80b564ce-17a9-4594-b542-7f7513d0b000 service cinder] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:15:44.176013 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80b564ce-17a9-4594-b542-7f7513d0b000 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:15:44.194967 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80b564ce-17a9-4594-b542-7f7513d0b000 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:15:44.197386 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80b564ce-17a9-4594-b542-7f7513d0b000 service cinder] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:15:44.230072 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1149/2310] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:15:44 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 71 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:15:53.955416 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=10 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:15:54.227119 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=44 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:16:06.153687 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72ccd051-4973-4c99-bd2b-707b277b4bee service cinder] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:16:06.174748 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72ccd051-4973-4c99-bd2b-707b277b4bee service cinder] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:16:06.178730 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bfb139a-35c8-4c3e-b35e-6a8fc5ae2e28 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:16:06.179350 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bfb139a-35c8-4c3e-b35e-6a8fc5ae2e28 service neutron] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:16:06.179620 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bfb139a-35c8-4c3e-b35e-6a8fc5ae2e28 service neutron] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:16:06.179888 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bfb139a-35c8-4c3e-b35e-6a8fc5ae2e28 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:16:06.187832 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bfb139a-35c8-4c3e-b35e-6a8fc5ae2e28 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:16:06.189307 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bfb139a-35c8-4c3e-b35e-6a8fc5ae2e28 service neutron] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:16:06.198318 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1162/2311] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:16:06 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:16:16.194349 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=10 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:16:41.993537 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80b564ce-17a9-4594-b542-7f7513d0b000 service cinder] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:16:42.003912 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80b564ce-17a9-4594-b542-7f7513d0b000 service cinder] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:16:42.009649 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77ec5948-a4c8-461a-ac29-0003952c0357 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:16:42.010451 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77ec5948-a4c8-461a-ac29-0003952c0357 service cinder] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:16:42.010913 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77ec5948-a4c8-461a-ac29-0003952c0357 service cinder] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:16:42.011284 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77ec5948-a4c8-461a-ac29-0003952c0357 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:16:42.022454 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77ec5948-a4c8-461a-ac29-0003952c0357 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:16:42.023987 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77ec5948-a4c8-461a-ac29-0003952c0357 service cinder] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:16:42.034025 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1150/2312] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:16:41 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:16:42.581846 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bfb139a-35c8-4c3e-b35e-6a8fc5ae2e28 service neutron] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:16:42.592817 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bfb139a-35c8-4c3e-b35e-6a8fc5ae2e28 service neutron] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:16:42.596719 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d42f9cfb-9191-466b-8649-b5eaf1aabae5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:16:42.597383 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d42f9cfb-9191-466b-8649-b5eaf1aabae5 service cinder] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:16:42.597657 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d42f9cfb-9191-466b-8649-b5eaf1aabae5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:16:42.597927 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d42f9cfb-9191-466b-8649-b5eaf1aabae5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:16:42.608471 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d42f9cfb-9191-466b-8649-b5eaf1aabae5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:16:42.610692 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d42f9cfb-9191-466b-8649-b5eaf1aabae5 service cinder] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:16:42.626503 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1163/2313] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:16:42 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:16:52.030183 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=6 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:16:52.617932 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=6 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:17:15.796416 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77ec5948-a4c8-461a-ac29-0003952c0357 service cinder] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:17:15.806133 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77ec5948-a4c8-461a-ac29-0003952c0357 service cinder] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:17:15.810453 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44f53f0f-40b8-4e18-89d4-6be268330b62 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:17:15.811063 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44f53f0f-40b8-4e18-89d4-6be268330b62 service neutron] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:17:15.811362 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44f53f0f-40b8-4e18-89d4-6be268330b62 service neutron] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:17:15.811632 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44f53f0f-40b8-4e18-89d4-6be268330b62 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:17:15.823946 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44f53f0f-40b8-4e18-89d4-6be268330b62 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:17:15.825449 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44f53f0f-40b8-4e18-89d4-6be268330b62 service neutron] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:17:15.839677 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1151/2314] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:17:15 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:17:17.087325 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d42f9cfb-9191-466b-8649-b5eaf1aabae5 service cinder] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:17:17.103611 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d42f9cfb-9191-466b-8649-b5eaf1aabae5 service cinder] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:17:17.107561 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af5dd498-bdcf-4678-b13e-75a4024db17f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:17:17.108172 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af5dd498-bdcf-4678-b13e-75a4024db17f service neutron] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:17:17.108450 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af5dd498-bdcf-4678-b13e-75a4024db17f service neutron] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:17:17.108717 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af5dd498-bdcf-4678-b13e-75a4024db17f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:17:17.111283 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-af5dd498-bdcf-4678-b13e-75a4024db17f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:17:17.141826 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af5dd498-bdcf-4678-b13e-75a4024db17f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:17:17.143441 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af5dd498-bdcf-4678-b13e-75a4024db17f service neutron] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:17:17.161269 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1164/2315] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:17:17 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 75 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:17:17.497188 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44f53f0f-40b8-4e18-89d4-6be268330b62 service neutron] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:17:17.498924 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-44f53f0f-40b8-4e18-89d4-6be268330b62 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:17:17.511021 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44f53f0f-40b8-4e18-89d4-6be268330b62 service neutron] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:17:17.515527 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db65ed2c-1015-42bb-8248-9c31bbbd9b44 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:17:17.516130 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db65ed2c-1015-42bb-8248-9c31bbbd9b44 service neutron] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:17:17.516386 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db65ed2c-1015-42bb-8248-9c31bbbd9b44 service neutron] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:17:17.516646 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db65ed2c-1015-42bb-8248-9c31bbbd9b44 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:17:17.518558 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-db65ed2c-1015-42bb-8248-9c31bbbd9b44 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:17:17.615506 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db65ed2c-1015-42bb-8248-9c31bbbd9b44 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:17:17.616988 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db65ed2c-1015-42bb-8248-9c31bbbd9b44 service neutron] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:17:17.636932 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1152/2316] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:17:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 140 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:17:21.563563 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af5dd498-bdcf-4678-b13e-75a4024db17f service neutron] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:17:21.575620 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af5dd498-bdcf-4678-b13e-75a4024db17f service neutron] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:17:21.581001 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68d97e02-db0c-4ef6-b1b5-ac2283e7ac52 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2a3834fa6894dada3291e4781bbd553', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:17:21.581704 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68d97e02-db0c-4ef6-b1b5-ac2283e7ac52 service placement] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:17:21.582070 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68d97e02-db0c-4ef6-b1b5-ac2283e7ac52 service placement] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:17:21.582385 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68d97e02-db0c-4ef6-b1b5-ac2283e7ac52 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:17:21.584501 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-68d97e02-db0c-4ef6-b1b5-ac2283e7ac52 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:17:21.596663 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68d97e02-db0c-4ef6-b1b5-ac2283e7ac52 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:17:21.598304 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68d97e02-db0c-4ef6-b1b5-ac2283e7ac52 service placement] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:17:21.612219 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1165/2317] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:17:21 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 49 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:17:24.730679 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db65ed2c-1015-42bb-8248-9c31bbbd9b44 service neutron] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:17:24.743055 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db65ed2c-1015-42bb-8248-9c31bbbd9b44 service neutron] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:17:24.748311 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12df45cd-ef21-4c25-951a-4d995bc02043 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2a3834fa6894dada3291e4781bbd553', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:17:24.749079 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12df45cd-ef21-4c25-951a-4d995bc02043 service placement] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:17:24.749536 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12df45cd-ef21-4c25-951a-4d995bc02043 service placement] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:17:24.749844 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12df45cd-ef21-4c25-951a-4d995bc02043 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:17:24.751703 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-12df45cd-ef21-4c25-951a-4d995bc02043 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:17:24.763732 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12df45cd-ef21-4c25-951a-4d995bc02043 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:17:24.765457 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12df45cd-ef21-4c25-951a-4d995bc02043 service placement] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:17:24.776188 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1153/2318] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:17:24 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 28 01:17:24.795401 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68d97e02-db0c-4ef6-b1b5-ac2283e7ac52 service placement] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:17:24.797125 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-68d97e02-db0c-4ef6-b1b5-ac2283e7ac52 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:17:24.811182 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68d97e02-db0c-4ef6-b1b5-ac2283e7ac52 service placement] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:17:24.816070 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fbcf979-bc86-4b3c-9efb-4ea14759dce2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:17:24.816790 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fbcf979-bc86-4b3c-9efb-4ea14759dce2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:17:24.817102 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fbcf979-bc86-4b3c-9efb-4ea14759dce2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:17:24.817710 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fbcf979-bc86-4b3c-9efb-4ea14759dce2 admin admin] PATH_INFO: `/v3/users/cac1ab72dcee40b19bcff4dd5a75d82d` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:17:24.819652 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fbcf979-bc86-4b3c-9efb-4ea14759dce2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=cac1ab72dcee40b19bcff4dd5a75d82d)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:17:24.821703 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fbcf979-bc86-4b3c-9efb-4ea14759dce2 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:17:24.876400 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-6fbcf979-bc86-4b3c-9efb-4ea14759dce2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cac1ab72dcee40b19bcff4dd5a75d82d'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:17:24.877131 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-6fbcf979-bc86-4b3c-9efb-4ea14759dce2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cac1ab72dcee40b19bcff4dd5a75d82d'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:17:24.885931 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-6fbcf979-bc86-4b3c-9efb-4ea14759dce2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cac1ab72dcee40b19bcff4dd5a75d82d'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:17:24.893751 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-6fbcf979-bc86-4b3c-9efb-4ea14759dce2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cac1ab72dcee40b19bcff4dd5a75d82d'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:17:24.894480 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-6fbcf979-bc86-4b3c-9efb-4ea14759dce2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cac1ab72dcee40b19bcff4dd5a75d82d'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:17:24.912764 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1166/2319] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 01:17:24 2022] DELETE /identity/v3/users/cac1ab72dcee40b19bcff4dd5a75d82d => generated 0 bytes in 118 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:17:25.355981 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12df45cd-ef21-4c25-951a-4d995bc02043 service placement] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:17:25.378772 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12df45cd-ef21-4c25-951a-4d995bc02043 service placement] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:17:25.465564 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a488ae2-c4f6-43e8-be9f-96ba3cd180d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:17:25.466575 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a488ae2-c4f6-43e8-be9f-96ba3cd180d3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:17:25.466575 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a488ae2-c4f6-43e8-be9f-96ba3cd180d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:17:25.466763 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a488ae2-c4f6-43e8-be9f-96ba3cd180d3 admin admin] PATH_INFO: `/v3/projects/287c855ef8e241978a4d1e7cb3c7a16f` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:17:25.467957 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a488ae2-c4f6-43e8-be9f-96ba3cd180d3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=287c855ef8e241978a4d1e7cb3c7a16f)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:17:25.470158 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a488ae2-c4f6-43e8-be9f-96ba3cd180d3 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:17:25.515665 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:17:25.523056 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-6a488ae2-c4f6-43e8-be9f-96ba3cd180d3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '287c855ef8e241978a4d1e7cb3c7a16f', 'request_id': 'req-6a488ae2-c4f6-43e8-be9f-96ba3cd180d3'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:17:25.528767 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-6a488ae2-c4f6-43e8-be9f-96ba3cd180d3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '287c855ef8e241978a4d1e7cb3c7a16f', 'request_id': 'req-6a488ae2-c4f6-43e8-be9f-96ba3cd180d3'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:17:25.535930 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-6a488ae2-c4f6-43e8-be9f-96ba3cd180d3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '287c855ef8e241978a4d1e7cb3c7a16f', 'request_id': 'req-6a488ae2-c4f6-43e8-be9f-96ba3cd180d3'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:17:25.551063 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-6a488ae2-c4f6-43e8-be9f-96ba3cd180d3 admin admin] The token cache is being invalidate because project 287c855ef8e241978a4d1e7cb3c7a16f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81507) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 01:17:25.551402 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-6a488ae2-c4f6-43e8-be9f-96ba3cd180d3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:17:25.551991 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-6a488ae2-c4f6-43e8-be9f-96ba3cd180d3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:17:25.554461 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1154/2320] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 01:17:25 2022] DELETE /identity/v3/projects/287c855ef8e241978a4d1e7cb3c7a16f => generated 0 bytes in 198 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:17:25.558280 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fbcf979-bc86-4b3c-9efb-4ea14759dce2 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:17:25.560631 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-6fbcf979-bc86-4b3c-9efb-4ea14759dce2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:17:25.583335 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fbcf979-bc86-4b3c-9efb-4ea14759dce2 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:17:25.670033 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dd61ecc-8d5f-4c83-a679-f4da710e054e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:17:25.670676 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd61ecc-8d5f-4c83-a679-f4da710e054e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:17:25.670999 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd61ecc-8d5f-4c83-a679-f4da710e054e admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:17:25.671254 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd61ecc-8d5f-4c83-a679-f4da710e054e admin admin] PATH_INFO: `/v3/users/4cac7d604715464d85da907d1dfc3482` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:17:25.687782 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dd61ecc-8d5f-4c83-a679-f4da710e054e admin admin] RBAC: Authorizing `identity:delete_user(user_id=4cac7d604715464d85da907d1dfc3482)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:17:25.690729 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dd61ecc-8d5f-4c83-a679-f4da710e054e admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:17:25.740297 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=84,keystone:DELETE=15,keystone:INSERT=1 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:17:25.751398 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-2dd61ecc-8d5f-4c83-a679-f4da710e054e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4cac7d604715464d85da907d1dfc3482'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:17:25.752167 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-2dd61ecc-8d5f-4c83-a679-f4da710e054e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4cac7d604715464d85da907d1dfc3482'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:17:25.761526 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-2dd61ecc-8d5f-4c83-a679-f4da710e054e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4cac7d604715464d85da907d1dfc3482'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:17:25.768770 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-2dd61ecc-8d5f-4c83-a679-f4da710e054e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4cac7d604715464d85da907d1dfc3482'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:17:25.769315 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-2dd61ecc-8d5f-4c83-a679-f4da710e054e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4cac7d604715464d85da907d1dfc3482'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:17:25.787227 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1167/2321] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 01:17:25 2022] DELETE /identity/v3/users/4cac7d604715464d85da907d1dfc3482 => generated 0 bytes in 230 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:17:25.978697 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a488ae2-c4f6-43e8-be9f-96ba3cd180d3 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:17:25.980141 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-6a488ae2-c4f6-43e8-be9f-96ba3cd180d3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:17:25.993898 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a488ae2-c4f6-43e8-be9f-96ba3cd180d3 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:17:26.054404 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-460db697-d609-413d-9ef4-95de92a0652e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:17:26.055008 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-460db697-d609-413d-9ef4-95de92a0652e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:17:26.055319 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-460db697-d609-413d-9ef4-95de92a0652e admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:17:26.055591 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-460db697-d609-413d-9ef4-95de92a0652e admin admin] PATH_INFO: `/v3/projects/7a69dc53c1a648c7a764c8ed1a1aeac7` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:17:26.066358 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-460db697-d609-413d-9ef4-95de92a0652e admin admin] RBAC: Authorizing `identity:delete_project(project_id=7a69dc53c1a648c7a764c8ed1a1aeac7)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:17:26.068633 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-460db697-d609-413d-9ef4-95de92a0652e admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:17:26.118053 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-460db697-d609-413d-9ef4-95de92a0652e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7a69dc53c1a648c7a764c8ed1a1aeac7', 'request_id': 'req-460db697-d609-413d-9ef4-95de92a0652e'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:17:26.123906 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-460db697-d609-413d-9ef4-95de92a0652e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7a69dc53c1a648c7a764c8ed1a1aeac7', 'request_id': 'req-460db697-d609-413d-9ef4-95de92a0652e'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:17:26.130856 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-460db697-d609-413d-9ef4-95de92a0652e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7a69dc53c1a648c7a764c8ed1a1aeac7', 'request_id': 'req-460db697-d609-413d-9ef4-95de92a0652e'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:17:26.147031 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-460db697-d609-413d-9ef4-95de92a0652e admin admin] The token cache is being invalidate because project 7a69dc53c1a648c7a764c8ed1a1aeac7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81507) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 01:17:26.147233 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-460db697-d609-413d-9ef4-95de92a0652e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:17:26.147905 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-460db697-d609-413d-9ef4-95de92a0652e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:17:26.149397 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1155/2322] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 01:17:25 2022] DELETE /identity/v3/projects/7a69dc53c1a648c7a764c8ed1a1aeac7 => generated 0 bytes in 171 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 28 01:17:35.783670 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=15,keystone:DELETE=5,keystone:INSERT=1 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:17:36.143382 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=64,keystone:DELETE=11,keystone:INSERT=2 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:19:15.830252 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dd61ecc-8d5f-4c83-a679-f4da710e054e admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:19:15.832250 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-2dd61ecc-8d5f-4c83-a679-f4da710e054e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:19:15.919915 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dd61ecc-8d5f-4c83-a679-f4da710e054e admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:19:15.990156 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea22201c-c26b-4104-9cd3-95ae14c51160 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2a3834fa6894dada3291e4781bbd553', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:19:15.990792 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea22201c-c26b-4104-9cd3-95ae14c51160 service placement] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:19:15.991061 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea22201c-c26b-4104-9cd3-95ae14c51160 service placement] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:19:15.991334 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea22201c-c26b-4104-9cd3-95ae14c51160 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:19:15.993149 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-ea22201c-c26b-4104-9cd3-95ae14c51160 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:19:16.005061 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea22201c-c26b-4104-9cd3-95ae14c51160 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:19:16.006657 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea22201c-c26b-4104-9cd3-95ae14c51160 service placement] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:19:16.017308 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1168/2323] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:19:15 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 188 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:19:17.102362 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-460db697-d609-413d-9ef4-95de92a0652e admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:19:17.103945 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-460db697-d609-413d-9ef4-95de92a0652e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:19:17.200782 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-460db697-d609-413d-9ef4-95de92a0652e admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:19:17.206606 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd4d1dc4-700d-4383-b909-7cc24f61de26 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '168b07df50c24593ba73dbde5b037aed', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:19:17.207456 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd4d1dc4-700d-4383-b909-7cc24f61de26 service nova] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:19:17.207812 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd4d1dc4-700d-4383-b909-7cc24f61de26 service nova] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:19:17.208642 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd4d1dc4-700d-4383-b909-7cc24f61de26 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:19:17.210112 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-fd4d1dc4-700d-4383-b909-7cc24f61de26 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:19:17.223430 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd4d1dc4-700d-4383-b909-7cc24f61de26 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:19:17.226052 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd4d1dc4-700d-4383-b909-7cc24f61de26 service nova] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:19:17.240490 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1156/2324] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:19:17 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 138 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:19:26.013294 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=58 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:19:27.233127 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=34 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:20:42.948239 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea22201c-c26b-4104-9cd3-95ae14c51160 service placement] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:20:42.950140 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-ea22201c-c26b-4104-9cd3-95ae14c51160 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:20:42.966364 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea22201c-c26b-4104-9cd3-95ae14c51160 service placement] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:20:42.973801 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ea64e92-34c7-4aff-8f6e-8bd92c9a0f59 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '168b07df50c24593ba73dbde5b037aed', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:20:42.974527 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ea64e92-34c7-4aff-8f6e-8bd92c9a0f59 service nova] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:20:42.974806 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ea64e92-34c7-4aff-8f6e-8bd92c9a0f59 service nova] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:20:42.975139 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ea64e92-34c7-4aff-8f6e-8bd92c9a0f59 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:20:42.977707 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-7ea64e92-34c7-4aff-8f6e-8bd92c9a0f59 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:20:43.040524 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ea64e92-34c7-4aff-8f6e-8bd92c9a0f59 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:20:43.042240 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ea64e92-34c7-4aff-8f6e-8bd92c9a0f59 service nova] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:20:43.053395 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1169/2325] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:20:42 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:20:44.125310 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd4d1dc4-700d-4383-b909-7cc24f61de26 service nova] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:20:44.138473 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd4d1dc4-700d-4383-b909-7cc24f61de26 service nova] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:20:44.145746 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87959370-7a84-4075-93c5-187e52add08d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '168b07df50c24593ba73dbde5b037aed', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:20:44.146367 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87959370-7a84-4075-93c5-187e52add08d service nova] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:20:44.146639 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87959370-7a84-4075-93c5-187e52add08d service nova] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:20:44.146905 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87959370-7a84-4075-93c5-187e52add08d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:20:44.148733 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-87959370-7a84-4075-93c5-187e52add08d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:20:44.224656 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87959370-7a84-4075-93c5-187e52add08d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:20:44.226287 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87959370-7a84-4075-93c5-187e52add08d service nova] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:20:44.236155 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1157/2326] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:20:44 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 111 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:20:44.254488 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ea64e92-34c7-4aff-8f6e-8bd92c9a0f59 service nova] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:20:44.257091 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-7ea64e92-34c7-4aff-8f6e-8bd92c9a0f59 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:20:44.338134 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ea64e92-34c7-4aff-8f6e-8bd92c9a0f59 service nova] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:20:44.351654 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-398c04c0-3ea3-4642-8664-51a99926e9bf service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:20:44.352296 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-398c04c0-3ea3-4642-8664-51a99926e9bf service cinder] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:20:44.352583 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-398c04c0-3ea3-4642-8664-51a99926e9bf service cinder] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:20:44.352857 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-398c04c0-3ea3-4642-8664-51a99926e9bf service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:20:44.354727 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-398c04c0-3ea3-4642-8664-51a99926e9bf service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:20:44.438629 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-398c04c0-3ea3-4642-8664-51a99926e9bf service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:20:44.440780 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-398c04c0-3ea3-4642-8664-51a99926e9bf service cinder] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:20:44.452832 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1170/2327] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:20:44 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 199 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:20:54.231351 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=33 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:20:54.447554 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=86 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:21:12.175134 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87959370-7a84-4075-93c5-187e52add08d service nova] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:21:12.177300 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-87959370-7a84-4075-93c5-187e52add08d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:21:12.249578 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87959370-7a84-4075-93c5-187e52add08d service nova] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:21:12.254105 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6debf6f2-8e16-4eba-840f-7b2f6b5dfa6e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:21:12.254725 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6debf6f2-8e16-4eba-840f-7b2f6b5dfa6e service neutron] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:21:12.255004 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6debf6f2-8e16-4eba-840f-7b2f6b5dfa6e service neutron] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:21:12.255328 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6debf6f2-8e16-4eba-840f-7b2f6b5dfa6e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:21:12.257121 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-6debf6f2-8e16-4eba-840f-7b2f6b5dfa6e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:21:12.268071 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6debf6f2-8e16-4eba-840f-7b2f6b5dfa6e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:21:12.269675 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6debf6f2-8e16-4eba-840f-7b2f6b5dfa6e service neutron] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:21:12.278012 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1158/2328] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:21:12 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:21:22.275122 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=31 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:21:42.214575 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-398c04c0-3ea3-4642-8664-51a99926e9bf service cinder] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:21:42.234181 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-398c04c0-3ea3-4642-8664-51a99926e9bf service cinder] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:21:42.240316 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a23f01ef-7782-466e-bc26-15c6f247b736 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '168b07df50c24593ba73dbde5b037aed', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:21:42.240930 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a23f01ef-7782-466e-bc26-15c6f247b736 service nova] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:21:42.241236 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a23f01ef-7782-466e-bc26-15c6f247b736 service nova] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:21:42.241504 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a23f01ef-7782-466e-bc26-15c6f247b736 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:21:42.243473 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-a23f01ef-7782-466e-bc26-15c6f247b736 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:21:42.322896 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a23f01ef-7782-466e-bc26-15c6f247b736 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:21:42.324623 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a23f01ef-7782-466e-bc26-15c6f247b736 service nova] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:21:42.343699 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1171/2329] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:21:42 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 130 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:21:43.237047 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6debf6f2-8e16-4eba-840f-7b2f6b5dfa6e service neutron] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:21:43.250921 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6debf6f2-8e16-4eba-840f-7b2f6b5dfa6e service neutron] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:21:43.257434 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b73f8d91-c9c1-432f-97bd-8847f763a4fc service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '168b07df50c24593ba73dbde5b037aed', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:21:43.258032 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b73f8d91-c9c1-432f-97bd-8847f763a4fc service nova] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:21:43.258315 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b73f8d91-c9c1-432f-97bd-8847f763a4fc service nova] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:21:43.258604 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b73f8d91-c9c1-432f-97bd-8847f763a4fc service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:21:43.260445 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-b73f8d91-c9c1-432f-97bd-8847f763a4fc service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:21:43.350282 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b73f8d91-c9c1-432f-97bd-8847f763a4fc service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:21:43.352055 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b73f8d91-c9c1-432f-97bd-8847f763a4fc service nova] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:21:43.372097 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1159/2330] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:21:43 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 135 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:21:52.340887 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=41 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:21:53.368500 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=41 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:22:15.848073 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a23f01ef-7782-466e-bc26-15c6f247b736 service nova] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:22:15.858392 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a23f01ef-7782-466e-bc26-15c6f247b736 service nova] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:22:15.864023 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-185b136b-02a4-4d27-ae47-b89675bdaf41 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:22:15.864667 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-185b136b-02a4-4d27-ae47-b89675bdaf41 service neutron] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:22:15.864945 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-185b136b-02a4-4d27-ae47-b89675bdaf41 service neutron] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:22:15.865327 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-185b136b-02a4-4d27-ae47-b89675bdaf41 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:22:15.867178 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-185b136b-02a4-4d27-ae47-b89675bdaf41 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:22:15.895918 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-185b136b-02a4-4d27-ae47-b89675bdaf41 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:22:15.897534 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-185b136b-02a4-4d27-ae47-b89675bdaf41 service neutron] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:22:15.913284 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1172/2331] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:22:15 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 66 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:22:16.895254 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b73f8d91-c9c1-432f-97bd-8847f763a4fc service nova] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:22:16.906243 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b73f8d91-c9c1-432f-97bd-8847f763a4fc service nova] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:22:16.910713 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d643ec4-b838-4aac-8bdc-eea050f93054 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:22:16.911481 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d643ec4-b838-4aac-8bdc-eea050f93054 service neutron] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:22:16.911651 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d643ec4-b838-4aac-8bdc-eea050f93054 service neutron] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:22:16.911863 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d643ec4-b838-4aac-8bdc-eea050f93054 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:22:16.913727 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-7d643ec4-b838-4aac-8bdc-eea050f93054 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:22:16.929212 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d643ec4-b838-4aac-8bdc-eea050f93054 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:22:16.930693 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d643ec4-b838-4aac-8bdc-eea050f93054 service neutron] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:22:16.943316 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1160/2332] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:22:16 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 49 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:22:25.907304 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=11 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:22:26.937529 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=6 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:23:18.672246 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-185b136b-02a4-4d27-ae47-b89675bdaf41 service neutron] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:23:18.697754 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-185b136b-02a4-4d27-ae47-b89675bdaf41 service neutron] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:23:18.709908 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96cfe022-11b7-4bb2-bf4b-9dd358f003ee service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '168b07df50c24593ba73dbde5b037aed', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:23:18.711487 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96cfe022-11b7-4bb2-bf4b-9dd358f003ee service nova] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:23:18.712428 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96cfe022-11b7-4bb2-bf4b-9dd358f003ee service nova] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:23:18.712428 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96cfe022-11b7-4bb2-bf4b-9dd358f003ee service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:23:18.715325 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-96cfe022-11b7-4bb2-bf4b-9dd358f003ee service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:23:18.734650 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96cfe022-11b7-4bb2-bf4b-9dd358f003ee service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:23:18.736294 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96cfe022-11b7-4bb2-bf4b-9dd358f003ee service nova] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:23:18.748030 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1173/2333] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:23:18 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 79 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:23:20.492046 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fd9c892-1bbe-4f61-9544-101cab478237 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:23:20.492624 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fd9c892-1bbe-4f61-9544-101cab478237 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:23:20.492624 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fd9c892-1bbe-4f61-9544-101cab478237 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:23:20.524109 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-6fd9c892-1bbe-4f61-9544-101cab478237 None None] MFA Rules not processed for user `d209174d0db04b3584556824c6d0a3d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:23:20.525839 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-6fd9c892-1bbe-4f61-9544-101cab478237 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:23:20.532768 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1161/2334] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:23:20 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:23:21.167300 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96cfe022-11b7-4bb2-bf4b-9dd358f003ee service nova] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:23:21.179207 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96cfe022-11b7-4bb2-bf4b-9dd358f003ee service nova] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:23:21.184267 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df1d6c41-c045-4579-8d35-50ecd69378d0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:23:21.185182 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df1d6c41-c045-4579-8d35-50ecd69378d0 service neutron] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:23:21.185528 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df1d6c41-c045-4579-8d35-50ecd69378d0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:23:21.185869 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df1d6c41-c045-4579-8d35-50ecd69378d0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:23:21.188362 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-df1d6c41-c045-4579-8d35-50ecd69378d0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:23:21.199446 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df1d6c41-c045-4579-8d35-50ecd69378d0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:23:21.201044 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df1d6c41-c045-4579-8d35-50ecd69378d0 service neutron] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:23:21.210361 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1174/2335] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:23:21 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:23:21.429695 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3897c416-60f2-4c31-b384-aaa6bbbc04ad None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:23:21.430396 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3897c416-60f2-4c31-b384-aaa6bbbc04ad None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:23:21.430906 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3897c416-60f2-4c31-b384-aaa6bbbc04ad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:23:21.487176 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-3897c416-60f2-4c31-b384-aaa6bbbc04ad None None] MFA Rules not processed for user `4894ef8b630e4bc084799c26096f7c66`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:23:21.489342 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-3897c416-60f2-4c31-b384-aaa6bbbc04ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:23:21.501541 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1162/2336] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:23:21 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 75 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:23:21.684158 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df1d6c41-c045-4579-8d35-50ecd69378d0 service neutron] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:23:21.685710 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-df1d6c41-c045-4579-8d35-50ecd69378d0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:23:21.704542 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df1d6c41-c045-4579-8d35-50ecd69378d0 service neutron] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:23:21.709707 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f075909-dbb3-4fc1-842f-1d648b1b098f service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2a3834fa6894dada3291e4781bbd553', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:23:21.710451 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f075909-dbb3-4fc1-842f-1d648b1b098f service placement] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:23:21.710613 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f075909-dbb3-4fc1-842f-1d648b1b098f service placement] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:23:21.711899 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f075909-dbb3-4fc1-842f-1d648b1b098f service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:23:21.714148 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-0f075909-dbb3-4fc1-842f-1d648b1b098f service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:23:21.729683 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f075909-dbb3-4fc1-842f-1d648b1b098f service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:23:21.732332 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f075909-dbb3-4fc1-842f-1d648b1b098f service placement] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:23:21.747643 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1175/2337] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:23:21 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 64 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:23:21.900501 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3897c416-60f2-4c31-b384-aaa6bbbc04ad None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:23:21.915225 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3897c416-60f2-4c31-b384-aaa6bbbc04ad None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:23:21.919016 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c42660cd-fd92-48fc-8229-d28fc07f192e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:23:21.919641 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c42660cd-fd92-48fc-8229-d28fc07f192e service neutron] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:23:21.919921 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c42660cd-fd92-48fc-8229-d28fc07f192e service neutron] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:23:21.920190 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c42660cd-fd92-48fc-8229-d28fc07f192e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:23:21.924904 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-c42660cd-fd92-48fc-8229-d28fc07f192e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:23:21.937206 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c42660cd-fd92-48fc-8229-d28fc07f192e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:23:21.938667 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c42660cd-fd92-48fc-8229-d28fc07f192e service neutron] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:23:21.947918 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1163/2338] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:23:21 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 49 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:23:31.741870 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=18 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:23:31.944062 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=18 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:23:31.988095 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f075909-dbb3-4fc1-842f-1d648b1b098f service placement] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:23:32.002256 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f075909-dbb3-4fc1-842f-1d648b1b098f service placement] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:23:32.010309 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e73d27ce-66cf-4357-bf06-08e33b857d98 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '168b07df50c24593ba73dbde5b037aed', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:23:32.011254 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e73d27ce-66cf-4357-bf06-08e33b857d98 service nova] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:23:32.011563 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e73d27ce-66cf-4357-bf06-08e33b857d98 service nova] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:23:32.011810 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e73d27ce-66cf-4357-bf06-08e33b857d98 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:23:32.013832 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-e73d27ce-66cf-4357-bf06-08e33b857d98 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:23:32.027708 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e73d27ce-66cf-4357-bf06-08e33b857d98 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:23:32.029426 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e73d27ce-66cf-4357-bf06-08e33b857d98 service nova] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:23:32.041054 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1176/2339] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:23:31 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 53 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:23:42.036823 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=6 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:24:18.418958 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c42660cd-fd92-48fc-8229-d28fc07f192e service neutron] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:24:18.461449 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c42660cd-fd92-48fc-8229-d28fc07f192e service neutron] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:24:18.466908 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-668724e3-9b40-4acf-81dc-f597d3710929 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2a3834fa6894dada3291e4781bbd553', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:24:18.467530 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-668724e3-9b40-4acf-81dc-f597d3710929 service placement] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:24:18.467813 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-668724e3-9b40-4acf-81dc-f597d3710929 service placement] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:24:18.468082 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-668724e3-9b40-4acf-81dc-f597d3710929 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:24:18.477607 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-668724e3-9b40-4acf-81dc-f597d3710929 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:24:18.479216 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-668724e3-9b40-4acf-81dc-f597d3710929 service placement] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:24:18.489758 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1164/2340] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:24:18 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 71 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 28 01:24:28.485203 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=16 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:25:16.304272 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0be9d5b-ace4-413b-917c-5744c57926f2 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:25:16.304825 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0be9d5b-ace4-413b-917c-5744c57926f2 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:25:16.304825 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0be9d5b-ace4-413b-917c-5744c57926f2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:25:16.344266 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-b0be9d5b-ace4-413b-917c-5744c57926f2 None None] MFA Rules not processed for user `3de5d68f67784e33bdcd19a9cdfed744`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:25:16.345413 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-b0be9d5b-ace4-413b-917c-5744c57926f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:25:16.352671 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1177/2341] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:25:16 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 50 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:25:20.410480 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-668724e3-9b40-4acf-81dc-f597d3710929 service placement] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:25:20.423405 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-668724e3-9b40-4acf-81dc-f597d3710929 service placement] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:25:20.432444 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-354d62f0-d56d-45d6-aa52-a600268b4c80 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '168b07df50c24593ba73dbde5b037aed', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:25:20.433250 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-354d62f0-d56d-45d6-aa52-a600268b4c80 service nova] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:25:20.433644 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-354d62f0-d56d-45d6-aa52-a600268b4c80 service nova] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:25:20.433923 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-354d62f0-d56d-45d6-aa52-a600268b4c80 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:25:20.445018 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-354d62f0-d56d-45d6-aa52-a600268b4c80 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:25:20.446735 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-354d62f0-d56d-45d6-aa52-a600268b4c80 service nova] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:25:20.459375 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1165/2342] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:25:20 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 48 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:25:26.341488 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=10 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:25:30.453456 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=6 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:25:44.616453 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0be9d5b-ace4-413b-917c-5744c57926f2 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:25:44.679307 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0be9d5b-ace4-413b-917c-5744c57926f2 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:25:44.685839 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2c30237-42ad-4931-894f-99e1ed98d520 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '168b07df50c24593ba73dbde5b037aed', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:25:44.686485 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2c30237-42ad-4931-894f-99e1ed98d520 service nova] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:25:44.686762 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2c30237-42ad-4931-894f-99e1ed98d520 service nova] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:25:44.687030 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2c30237-42ad-4931-894f-99e1ed98d520 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:25:44.696830 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2c30237-42ad-4931-894f-99e1ed98d520 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:25:44.698585 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2c30237-42ad-4931-894f-99e1ed98d520 service nova] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:25:44.731114 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1178/2343] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:25:44 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 115 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:25:52.490657 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-354d62f0-d56d-45d6-aa52-a600268b4c80 service nova] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:25:52.548243 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-354d62f0-d56d-45d6-aa52-a600268b4c80 service nova] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:25:52.554481 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93e9d848-d217-4e2d-9cdb-bc046b57b504 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '168b07df50c24593ba73dbde5b037aed', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:25:52.555113 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93e9d848-d217-4e2d-9cdb-bc046b57b504 service nova] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:25:52.555392 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93e9d848-d217-4e2d-9cdb-bc046b57b504 service nova] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:25:52.555670 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93e9d848-d217-4e2d-9cdb-bc046b57b504 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:25:52.566951 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93e9d848-d217-4e2d-9cdb-bc046b57b504 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:25:52.568595 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93e9d848-d217-4e2d-9cdb-bc046b57b504 service nova] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:25:52.597243 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1166/2344] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:25:52 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:25:54.728544 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=31 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:26:00.861260 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4d1d710-f5f5-4b25-bf60-a7de7f65af11 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:26:00.862259 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4d1d710-f5f5-4b25-bf60-a7de7f65af11 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:26:00.862734 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4d1d710-f5f5-4b25-bf60-a7de7f65af11 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:26:00.884220 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-a4d1d710-f5f5-4b25-bf60-a7de7f65af11 None None] MFA Rules not processed for user `d209174d0db04b3584556824c6d0a3d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:26:00.886189 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-a4d1d710-f5f5-4b25-bf60-a7de7f65af11 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:26:00.894053 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1179/2345] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:26:00 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 28 01:26:02.594575 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=31 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:26:10.879806 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=3 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:26:15.056037 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93e9d848-d217-4e2d-9cdb-bc046b57b504 service nova] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:26:15.081701 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93e9d848-d217-4e2d-9cdb-bc046b57b504 service nova] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:26:15.085326 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bf3c5c0-0e7e-42dd-b802-c3c855e5d15f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:26:15.086049 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bf3c5c0-0e7e-42dd-b802-c3c855e5d15f service neutron] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:26:15.086298 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bf3c5c0-0e7e-42dd-b802-c3c855e5d15f service neutron] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:26:15.086567 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bf3c5c0-0e7e-42dd-b802-c3c855e5d15f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:26:15.095096 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bf3c5c0-0e7e-42dd-b802-c3c855e5d15f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:26:15.096605 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bf3c5c0-0e7e-42dd-b802-c3c855e5d15f service neutron] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:26:15.106145 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1167/2346] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:26:15 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 51 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:26:25.102004 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=10 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:26:45.637111 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4d1d710-f5f5-4b25-bf60-a7de7f65af11 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:26:45.663106 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4d1d710-f5f5-4b25-bf60-a7de7f65af11 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:26:45.666876 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6473dac-e8e0-48de-b9f4-ee9e1fd3d161 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:26:45.667507 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6473dac-e8e0-48de-b9f4-ee9e1fd3d161 service cinder] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:26:45.667784 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6473dac-e8e0-48de-b9f4-ee9e1fd3d161 service cinder] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:26:45.668052 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6473dac-e8e0-48de-b9f4-ee9e1fd3d161 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:26:45.682068 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6473dac-e8e0-48de-b9f4-ee9e1fd3d161 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:26:45.683613 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6473dac-e8e0-48de-b9f4-ee9e1fd3d161 service cinder] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:26:45.697080 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1180/2347] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:26:45 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 60 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:26:45.944478 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bf3c5c0-0e7e-42dd-b802-c3c855e5d15f service neutron] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:26:45.956083 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bf3c5c0-0e7e-42dd-b802-c3c855e5d15f service neutron] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:26:45.961100 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c328c02f-c162-4751-94d4-314bbb4d1e36 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:26:45.961862 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c328c02f-c162-4751-94d4-314bbb4d1e36 service cinder] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:26:45.962203 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c328c02f-c162-4751-94d4-314bbb4d1e36 service cinder] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:26:45.962533 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c328c02f-c162-4751-94d4-314bbb4d1e36 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:26:45.973345 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c328c02f-c162-4751-94d4-314bbb4d1e36 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:26:45.974892 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c328c02f-c162-4751-94d4-314bbb4d1e36 service cinder] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:26:46.006173 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1168/2348] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:26:45 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 62 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:26:46.801311 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6473dac-e8e0-48de-b9f4-ee9e1fd3d161 service cinder] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:26:46.820047 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6473dac-e8e0-48de-b9f4-ee9e1fd3d161 service cinder] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:26:46.827387 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06cb2082-d200-4d03-b001-b224a02f7292 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '168b07df50c24593ba73dbde5b037aed', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:26:46.828063 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06cb2082-d200-4d03-b001-b224a02f7292 service nova] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:26:46.828425 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06cb2082-d200-4d03-b001-b224a02f7292 service nova] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:26:46.828701 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06cb2082-d200-4d03-b001-b224a02f7292 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:26:46.839626 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06cb2082-d200-4d03-b001-b224a02f7292 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:26:46.841507 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06cb2082-d200-4d03-b001-b224a02f7292 service nova] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:26:46.851744 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1181/2349] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:26:46 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 51 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:26:56.002592 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=13 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:26:56.847260 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=16 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:27:16.121000 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c328c02f-c162-4751-94d4-314bbb4d1e36 service cinder] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:27:16.130947 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c328c02f-c162-4751-94d4-314bbb4d1e36 service cinder] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:27:16.134924 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c44a5a7-ef94-44c4-92a2-60fdbf393b8c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:27:16.135530 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c44a5a7-ef94-44c4-92a2-60fdbf393b8c service neutron] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:27:16.135785 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c44a5a7-ef94-44c4-92a2-60fdbf393b8c service neutron] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:27:16.136051 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c44a5a7-ef94-44c4-92a2-60fdbf393b8c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:27:16.147213 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c44a5a7-ef94-44c4-92a2-60fdbf393b8c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:27:16.148690 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c44a5a7-ef94-44c4-92a2-60fdbf393b8c service neutron] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:27:16.160421 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1169/2350] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:27:16 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:27:17.345300 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06cb2082-d200-4d03-b001-b224a02f7292 service nova] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:27:17.355648 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06cb2082-d200-4d03-b001-b224a02f7292 service nova] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:27:17.360163 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adf77079-2df4-45b7-9881-a2910b90d3f0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:27:17.360769 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adf77079-2df4-45b7-9881-a2910b90d3f0 service neutron] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:27:17.361074 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adf77079-2df4-45b7-9881-a2910b90d3f0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:27:17.361342 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adf77079-2df4-45b7-9881-a2910b90d3f0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:27:17.373345 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adf77079-2df4-45b7-9881-a2910b90d3f0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:27:17.374731 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adf77079-2df4-45b7-9881-a2910b90d3f0 service neutron] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:27:17.386440 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1182/2351] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:27:17 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:27:26.155085 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=6 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:27:27.381028 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=6 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:29:19.139601 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c44a5a7-ef94-44c4-92a2-60fdbf393b8c service neutron] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:29:19.141367 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-5c44a5a7-ef94-44c4-92a2-60fdbf393b8c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:29:19.216883 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c44a5a7-ef94-44c4-92a2-60fdbf393b8c service neutron] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:29:19.222887 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddd4a08f-28be-4bd0-adee-2443b4c95d98 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '168b07df50c24593ba73dbde5b037aed', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:29:19.223470 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddd4a08f-28be-4bd0-adee-2443b4c95d98 service nova] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:29:19.224022 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddd4a08f-28be-4bd0-adee-2443b4c95d98 service nova] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:29:19.224022 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddd4a08f-28be-4bd0-adee-2443b4c95d98 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:29:19.233557 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddd4a08f-28be-4bd0-adee-2443b4c95d98 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:29:19.235215 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddd4a08f-28be-4bd0-adee-2443b4c95d98 service nova] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:29:19.245994 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1170/2352] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:29:19 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:29:21.546000 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adf77079-2df4-45b7-9881-a2910b90d3f0 service neutron] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:29:21.621879 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adf77079-2df4-45b7-9881-a2910b90d3f0 service neutron] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:29:21.627603 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23d8e359-c9a4-4a3f-81da-b93b9eba9fe5 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2a3834fa6894dada3291e4781bbd553', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:29:21.628211 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23d8e359-c9a4-4a3f-81da-b93b9eba9fe5 service placement] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:29:21.628488 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23d8e359-c9a4-4a3f-81da-b93b9eba9fe5 service placement] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:29:21.628754 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23d8e359-c9a4-4a3f-81da-b93b9eba9fe5 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:29:21.630795 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-23d8e359-c9a4-4a3f-81da-b93b9eba9fe5 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:29:21.656721 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23d8e359-c9a4-4a3f-81da-b93b9eba9fe5 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:29:21.658434 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23d8e359-c9a4-4a3f-81da-b93b9eba9fe5 service placement] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:29:21.668354 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1183/2353] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:29:21 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 123 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:29:29.241657 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=34 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:29:31.664243 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=37 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:30:52.684209 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddd4a08f-28be-4bd0-adee-2443b4c95d98 service nova] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:30:52.686273 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-ddd4a08f-28be-4bd0-adee-2443b4c95d98 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:30:52.763137 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddd4a08f-28be-4bd0-adee-2443b4c95d98 service nova] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:30:52.766893 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9794f3c-995c-4801-ab78-03745c295f3b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:30:52.767509 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9794f3c-995c-4801-ab78-03745c295f3b service cinder] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:30:52.767787 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9794f3c-995c-4801-ab78-03745c295f3b service cinder] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:30:52.768057 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9794f3c-995c-4801-ab78-03745c295f3b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:30:52.769962 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-e9794f3c-995c-4801-ab78-03745c295f3b service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:30:52.814949 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23d8e359-c9a4-4a3f-81da-b93b9eba9fe5 service placement] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:30:52.816668 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-23d8e359-c9a4-4a3f-81da-b93b9eba9fe5 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:30:52.831791 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23d8e359-c9a4-4a3f-81da-b93b9eba9fe5 service placement] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:30:52.836995 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37ac6614-9f9e-4b0b-b5bc-689f56f3d877 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:30:52.837764 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37ac6614-9f9e-4b0b-b5bc-689f56f3d877 service cinder] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:30:52.838042 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37ac6614-9f9e-4b0b-b5bc-689f56f3d877 service cinder] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:30:52.838312 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37ac6614-9f9e-4b0b-b5bc-689f56f3d877 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:30:52.841459 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-37ac6614-9f9e-4b0b-b5bc-689f56f3d877 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:30:52.856762 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9794f3c-995c-4801-ab78-03745c295f3b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:30:52.859681 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9794f3c-995c-4801-ab78-03745c295f3b service cinder] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:30:52.878066 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1171/2354] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:30:52 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 194 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:30:52.925676 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37ac6614-9f9e-4b0b-b5bc-689f56f3d877 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:30:52.927258 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37ac6614-9f9e-4b0b-b5bc-689f56f3d877 service cinder] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:30:52.938797 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1184/2355] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:30:52 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 124 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 28 01:30:53.212813 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9794f3c-995c-4801-ab78-03745c295f3b service cinder] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:30:53.226159 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9794f3c-995c-4801-ab78-03745c295f3b service cinder] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:30:53.231467 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84490515-3184-4d4c-aea8-0bfebf318c3d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:30:53.232218 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84490515-3184-4d4c-aea8-0bfebf318c3d service neutron] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:30:53.232570 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84490515-3184-4d4c-aea8-0bfebf318c3d service neutron] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:30:53.232934 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84490515-3184-4d4c-aea8-0bfebf318c3d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:30:53.235053 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-84490515-3184-4d4c-aea8-0bfebf318c3d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:30:53.357253 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84490515-3184-4d4c-aea8-0bfebf318c3d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:30:53.360689 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84490515-3184-4d4c-aea8-0bfebf318c3d service neutron] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:30:53.378414 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:30:53.388543 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1172/2356] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:30:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 178 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:30:56.337334 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37ac6614-9f9e-4b0b-b5bc-689f56f3d877 service cinder] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:30:56.364541 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37ac6614-9f9e-4b0b-b5bc-689f56f3d877 service cinder] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:30:56.370007 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef0f0648-bbbb-4f33-9ed7-5c6fc3a149e5 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '168b07df50c24593ba73dbde5b037aed', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:30:56.371854 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef0f0648-bbbb-4f33-9ed7-5c6fc3a149e5 service nova] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:30:56.375274 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef0f0648-bbbb-4f33-9ed7-5c6fc3a149e5 service nova] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:30:56.375274 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef0f0648-bbbb-4f33-9ed7-5c6fc3a149e5 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:30:56.376952 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-ef0f0648-bbbb-4f33-9ed7-5c6fc3a149e5 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:30:56.390227 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef0f0648-bbbb-4f33-9ed7-5c6fc3a149e5 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:30:56.391938 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef0f0648-bbbb-4f33-9ed7-5c6fc3a149e5 service nova] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:30:56.403307 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1185/2357] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:30:56 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 69 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:30:58.970889 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84490515-3184-4d4c-aea8-0bfebf318c3d service neutron] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:30:58.986317 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84490515-3184-4d4c-aea8-0bfebf318c3d service neutron] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:30:58.994472 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6f6aa3f-e75c-4fbd-a849-0051ec6041cc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:30:58.995233 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6f6aa3f-e75c-4fbd-a849-0051ec6041cc service neutron] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:30:58.995776 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6f6aa3f-e75c-4fbd-a849-0051ec6041cc service neutron] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:30:58.997267 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6f6aa3f-e75c-4fbd-a849-0051ec6041cc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:30:58.999404 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-c6f6aa3f-e75c-4fbd-a849-0051ec6041cc service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:30:59.013905 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6f6aa3f-e75c-4fbd-a849-0051ec6041cc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:30:59.016366 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6f6aa3f-e75c-4fbd-a849-0051ec6041cc service neutron] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:30:59.027856 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1173/2358] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:30:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 58 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:31:02.796231 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef0f0648-bbbb-4f33-9ed7-5c6fc3a149e5 service nova] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:31:02.798443 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-ef0f0648-bbbb-4f33-9ed7-5c6fc3a149e5 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:31:02.827016 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef0f0648-bbbb-4f33-9ed7-5c6fc3a149e5 service nova] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:31:02.839501 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7189b42-2cb5-422c-ac07-4245c3a47195 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2a3834fa6894dada3291e4781bbd553', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:31:02.842065 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7189b42-2cb5-422c-ac07-4245c3a47195 service placement] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:31:02.845086 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7189b42-2cb5-422c-ac07-4245c3a47195 service placement] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:31:02.845616 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7189b42-2cb5-422c-ac07-4245c3a47195 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:31:02.848729 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-b7189b42-2cb5-422c-ac07-4245c3a47195 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:31:02.866782 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6f6aa3f-e75c-4fbd-a849-0051ec6041cc service neutron] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:31:02.871420 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7189b42-2cb5-422c-ac07-4245c3a47195 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:31:02.871664 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-c6f6aa3f-e75c-4fbd-a849-0051ec6041cc service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:31:02.873626 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7189b42-2cb5-422c-ac07-4245c3a47195 service placement] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:31:02.894307 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6f6aa3f-e75c-4fbd-a849-0051ec6041cc service neutron] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:31:02.897683 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1186/2359] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:31:02 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:31:02.901222 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de6417e0-9bd5-4ba7-a833-d9ca4119dd9d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:31:02.902154 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de6417e0-9bd5-4ba7-a833-d9ca4119dd9d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:31:02.902474 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de6417e0-9bd5-4ba7-a833-d9ca4119dd9d admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:31:02.902746 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de6417e0-9bd5-4ba7-a833-d9ca4119dd9d admin admin] PATH_INFO: `/v3/users/4894ef8b630e4bc084799c26096f7c66` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:31:02.903927 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de6417e0-9bd5-4ba7-a833-d9ca4119dd9d admin admin] RBAC: Authorizing `identity:delete_user(user_id=4894ef8b630e4bc084799c26096f7c66)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:31:02.906160 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de6417e0-9bd5-4ba7-a833-d9ca4119dd9d admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:31:02.975894 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-de6417e0-9bd5-4ba7-a833-d9ca4119dd9d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4894ef8b630e4bc084799c26096f7c66'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:31:02.995166 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-de6417e0-9bd5-4ba7-a833-d9ca4119dd9d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4894ef8b630e4bc084799c26096f7c66'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:31:03.004193 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-de6417e0-9bd5-4ba7-a833-d9ca4119dd9d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4894ef8b630e4bc084799c26096f7c66'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:31:03.012572 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-de6417e0-9bd5-4ba7-a833-d9ca4119dd9d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4894ef8b630e4bc084799c26096f7c66'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:31:03.013420 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-de6417e0-9bd5-4ba7-a833-d9ca4119dd9d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4894ef8b630e4bc084799c26096f7c66'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:31:03.020762 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1174/2360] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 01:31:02 2022] DELETE /identity/v3/users/4894ef8b630e4bc084799c26096f7c66 => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:31:03.218692 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7189b42-2cb5-422c-ac07-4245c3a47195 service placement] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:31:03.246367 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7189b42-2cb5-422c-ac07-4245c3a47195 service placement] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:31:03.327683 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e5f5297-546e-4c26-9725-f4079e367e98 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:31:03.328327 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e5f5297-546e-4c26-9725-f4079e367e98 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:31:03.328603 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e5f5297-546e-4c26-9725-f4079e367e98 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:31:03.328875 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e5f5297-546e-4c26-9725-f4079e367e98 admin admin] PATH_INFO: `/v3/projects/19f069b9ad7a4a5984fa947ad755db0a` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:31:03.330102 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e5f5297-546e-4c26-9725-f4079e367e98 admin admin] RBAC: Authorizing `identity:delete_project(project_id=19f069b9ad7a4a5984fa947ad755db0a)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:31:03.332082 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e5f5297-546e-4c26-9725-f4079e367e98 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:31:03.390248 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-4e5f5297-546e-4c26-9725-f4079e367e98 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '19f069b9ad7a4a5984fa947ad755db0a', 'request_id': 'req-4e5f5297-546e-4c26-9725-f4079e367e98'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:31:03.398674 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-4e5f5297-546e-4c26-9725-f4079e367e98 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '19f069b9ad7a4a5984fa947ad755db0a', 'request_id': 'req-4e5f5297-546e-4c26-9725-f4079e367e98'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:31:03.408726 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-4e5f5297-546e-4c26-9725-f4079e367e98 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '19f069b9ad7a4a5984fa947ad755db0a', 'request_id': 'req-4e5f5297-546e-4c26-9725-f4079e367e98'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:31:03.411719 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=94,keystone:DELETE=6 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:31:03.426124 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-4e5f5297-546e-4c26-9725-f4079e367e98 admin admin] The token cache is being invalidate because project 19f069b9ad7a4a5984fa947ad755db0a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81508) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 01:31:03.426550 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-4e5f5297-546e-4c26-9725-f4079e367e98 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:31:03.427473 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-4e5f5297-546e-4c26-9725-f4079e367e98 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:31:03.429755 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1187/2361] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 01:31:03 2022] DELETE /identity/v3/projects/19f069b9ad7a4a5984fa947ad755db0a => generated 0 bytes in 211 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:31:03.435096 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf7f4733-8d6d-4422-a6a0-f21968fed595 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:31:03.435382 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf7f4733-8d6d-4422-a6a0-f21968fed595 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:31:03.435656 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf7f4733-8d6d-4422-a6a0-f21968fed595 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:31:03.504731 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-bf7f4733-8d6d-4422-a6a0-f21968fed595 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:31:03.506040 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-bf7f4733-8d6d-4422-a6a0-f21968fed595 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:31:03.571601 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1175/2362] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:31:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 28 01:31:03.580213 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f0ee9aa-9fe7-4c5d-97d2-681315ec9cef None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:31:03.580697 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f0ee9aa-9fe7-4c5d-97d2-681315ec9cef None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:31:03.581178 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f0ee9aa-9fe7-4c5d-97d2-681315ec9cef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:31:03.603632 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-2f0ee9aa-9fe7-4c5d-97d2-681315ec9cef None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:31:03.603632 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-2f0ee9aa-9fe7-4c5d-97d2-681315ec9cef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:31:03.634454 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1188/2363] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:31:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 55 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:31:03.639269 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf7f4733-8d6d-4422-a6a0-f21968fed595 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:31:03.654878 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf7f4733-8d6d-4422-a6a0-f21968fed595 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:31:03.660862 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff3178e8-4fb0-44b5-afe7-aef20508cf72 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:31:03.661596 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3178e8-4fb0-44b5-afe7-aef20508cf72 admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:31:03.661941 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3178e8-4fb0-44b5-afe7-aef20508cf72 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:31:03.662388 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3178e8-4fb0-44b5-afe7-aef20508cf72 admin admin] PATH_INFO: `/v3/domains` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:31:03.662950 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff3178e8-4fb0-44b5-afe7-aef20508cf72 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 01:31:03.663270 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff3178e8-4fb0-44b5-afe7-aef20508cf72 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:31:03.666448 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff3178e8-4fb0-44b5-afe7-aef20508cf72 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:31:03.682567 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1176/2364] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Aug 28 01:31:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 01:31:03.687359 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f0ee9aa-9fe7-4c5d-97d2-681315ec9cef None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:31:03.698127 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f0ee9aa-9fe7-4c5d-97d2-681315ec9cef None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:31:03.703588 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3952c76-51c9-46d7-97e3-3262da16d6ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:31:03.704219 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3952c76-51c9-46d7-97e3-3262da16d6ce admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:31:03.704504 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3952c76-51c9-46d7-97e3-3262da16d6ce admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:31:03.704815 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3952c76-51c9-46d7-97e3-3262da16d6ce admin admin] PATH_INFO: `/v3/projects` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:31:03.705506 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3952c76-51c9-46d7-97e3-3262da16d6ce admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:31:03.707677 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3952c76-51c9-46d7-97e3-3262da16d6ce admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:31:03.769107 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1189/2365] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 01:31:03 2022] POST /identity/v3/projects => generated 375 bytes in 79 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 01:31:03.771798 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff3178e8-4fb0-44b5-afe7-aef20508cf72 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:31:03.784330 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff3178e8-4fb0-44b5-afe7-aef20508cf72 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:31:03.794816 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:31:03.853714 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a52703a4-d6f4-4578-b6a0-b697709dcb43 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:31:03.854413 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a52703a4-d6f4-4578-b6a0-b697709dcb43 admin admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:31:03.854698 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a52703a4-d6f4-4578-b6a0-b697709dcb43 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:31:03.854972 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a52703a4-d6f4-4578-b6a0-b697709dcb43 admin admin] PATH_INFO: `/v3/users` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:31:03.855623 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a52703a4-d6f4-4578-b6a0-b697709dcb43 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:31:03.857617 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a52703a4-d6f4-4578-b6a0-b697709dcb43 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:31:03.884620 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1177/2366] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 01:31:03 2022] POST /identity/v3/users => generated 288 bytes in 114 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 01:31:03.888813 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3952c76-51c9-46d7-97e3-3262da16d6ce admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:31:03.900314 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3952c76-51c9-46d7-97e3-3262da16d6ce admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:31:03.905057 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6a199ed-2b43-4620-91eb-0f9cbffd8948 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:31:03.905668 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6a199ed-2b43-4620-91eb-0f9cbffd8948 admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:31:03.905945 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6a199ed-2b43-4620-91eb-0f9cbffd8948 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:31:03.906215 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6a199ed-2b43-4620-91eb-0f9cbffd8948 admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:31:03.906886 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6a199ed-2b43-4620-91eb-0f9cbffd8948 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:31:03.908578 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6a199ed-2b43-4620-91eb-0f9cbffd8948 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:31:03.919438 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1190/2367] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 01:31:03 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 28 01:31:03.923640 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a52703a4-d6f4-4578-b6a0-b697709dcb43 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:31:03.934166 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a52703a4-d6f4-4578-b6a0-b697709dcb43 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:31:03.939354 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf547235-9c16-47d1-bf8b-ada12ff2e46c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:31:03.939959 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf547235-9c16-47d1-bf8b-ada12ff2e46c admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:31:03.940240 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf547235-9c16-47d1-bf8b-ada12ff2e46c admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:31:03.940505 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf547235-9c16-47d1-bf8b-ada12ff2e46c admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:31:03.941309 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf547235-9c16-47d1-bf8b-ada12ff2e46c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:31:03.942968 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf547235-9c16-47d1-bf8b-ada12ff2e46c admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:31:03.953890 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1178/2368] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 01:31:03 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 01:31:03.958422 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6a199ed-2b43-4620-91eb-0f9cbffd8948 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:31:03.968565 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6a199ed-2b43-4620-91eb-0f9cbffd8948 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:31:03.973534 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d745b868-7a29-483a-a524-47f88491d113 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:31:03.974192 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d745b868-7a29-483a-a524-47f88491d113 admin admin] REQUEST_METHOD: `PUT` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:31:03.974473 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d745b868-7a29-483a-a524-47f88491d113 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:31:03.974745 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d745b868-7a29-483a-a524-47f88491d113 admin admin] PATH_INFO: `/v3/projects/765f789deef04206a3f89cfda8d1ee08/users/eebc2779346f4fdbbf3bb89027638c24/roles/0487ef7e943e4701915cb8ce0f2899d7` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:31:03.990418 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d745b868-7a29-483a-a524-47f88491d113 admin admin] RBAC: Authorizing `identity:create_grant(project_id=765f789deef04206a3f89cfda8d1ee08, user_id=eebc2779346f4fdbbf3bb89027638c24, role_id=0487ef7e943e4701915cb8ce0f2899d7)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:31:03.993130 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d745b868-7a29-483a-a524-47f88491d113 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:31:04.009209 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1191/2369] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 01:31:03 2022] PUT /identity/v3/projects/765f789deef04206a3f89cfda8d1ee08/users/eebc2779346f4fdbbf3bb89027638c24/roles/0487ef7e943e4701915cb8ce0f2899d7 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:31:04.019872 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfc60e36-ca20-4635-95e8-d04767aa13dc None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:31:04.020184 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfc60e36-ca20-4635-95e8-d04767aa13dc None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:31:04.020456 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfc60e36-ca20-4635-95e8-d04767aa13dc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:31:04.061794 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-cfc60e36-ca20-4635-95e8-d04767aa13dc None None] MFA Rules not processed for user `eebc2779346f4fdbbf3bb89027638c24`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:31:04.064062 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-cfc60e36-ca20-4635-95e8-d04767aa13dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:31:04.145003 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1179/2370] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:31:04 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 28 01:31:04.155806 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d745b868-7a29-483a-a524-47f88491d113 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:31:04.157661 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-d745b868-7a29-483a-a524-47f88491d113 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:31:04.237515 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d745b868-7a29-483a-a524-47f88491d113 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:31:04.243366 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c2a361c-a343-4cbe-a1a0-0419c48add2f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '168b07df50c24593ba73dbde5b037aed', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:31:04.243996 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c2a361c-a343-4cbe-a1a0-0419c48add2f service nova] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:31:04.244276 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c2a361c-a343-4cbe-a1a0-0419c48add2f service nova] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:31:04.244583 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c2a361c-a343-4cbe-a1a0-0419c48add2f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:31:04.253200 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c2a361c-a343-4cbe-a1a0-0419c48add2f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:31:04.254779 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c2a361c-a343-4cbe-a1a0-0419c48add2f service nova] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:31:04.263926 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1192/2371] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:31:04 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:31:04.501940 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfc60e36-ca20-4635-95e8-d04767aa13dc None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:31:04.575480 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfc60e36-ca20-4635-95e8-d04767aa13dc None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:31:04.580306 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b8b3c81-e433-46e9-a591-7aeb5f0da35f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:31:04.580940 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b8b3c81-e433-46e9-a591-7aeb5f0da35f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:31:04.581250 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b8b3c81-e433-46e9-a591-7aeb5f0da35f admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:31:04.581519 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b8b3c81-e433-46e9-a591-7aeb5f0da35f admin admin] PATH_INFO: `/v3/users/eebc2779346f4fdbbf3bb89027638c24` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:31:04.582659 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b8b3c81-e433-46e9-a591-7aeb5f0da35f admin admin] RBAC: Authorizing `identity:delete_user(user_id=eebc2779346f4fdbbf3bb89027638c24)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:31:04.584608 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b8b3c81-e433-46e9-a591-7aeb5f0da35f admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:31:04.608149 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=93,keystone:INSERT=3,keystone:DELETE=4 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:31:04.634905 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-8b8b3c81-e433-46e9-a591-7aeb5f0da35f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'eebc2779346f4fdbbf3bb89027638c24'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:31:04.650702 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-8b8b3c81-e433-46e9-a591-7aeb5f0da35f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'eebc2779346f4fdbbf3bb89027638c24'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:31:04.661538 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-8b8b3c81-e433-46e9-a591-7aeb5f0da35f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'eebc2779346f4fdbbf3bb89027638c24'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:31:04.672873 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-8b8b3c81-e433-46e9-a591-7aeb5f0da35f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'eebc2779346f4fdbbf3bb89027638c24'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:31:04.674027 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-8b8b3c81-e433-46e9-a591-7aeb5f0da35f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'eebc2779346f4fdbbf3bb89027638c24'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:31:04.684105 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1180/2372] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 01:31:04 2022] DELETE /identity/v3/users/eebc2779346f4fdbbf3bb89027638c24 => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:31:04.690910 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa1bd55c-44a9-4fef-8276-19023f33a267 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:31:04.691036 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa1bd55c-44a9-4fef-8276-19023f33a267 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:31:04.691376 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa1bd55c-44a9-4fef-8276-19023f33a267 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:31:04.711990 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-fa1bd55c-44a9-4fef-8276-19023f33a267 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:31:04.713262 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-fa1bd55c-44a9-4fef-8276-19023f33a267 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:31:04.781375 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1193/2373] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:31:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:31:04.791241 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b8b3c81-e433-46e9-a591-7aeb5f0da35f admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:31:04.792757 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-8b8b3c81-e433-46e9-a591-7aeb5f0da35f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:31:04.853385 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b8b3c81-e433-46e9-a591-7aeb5f0da35f admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:31:04.856984 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b7ff25c-797d-4a36-b79e-958f565d41cc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:31:04.857594 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b7ff25c-797d-4a36-b79e-958f565d41cc service neutron] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:31:04.857867 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b7ff25c-797d-4a36-b79e-958f565d41cc service neutron] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:31:04.858134 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b7ff25c-797d-4a36-b79e-958f565d41cc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:31:04.869309 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b7ff25c-797d-4a36-b79e-958f565d41cc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:31:04.870778 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b7ff25c-797d-4a36-b79e-958f565d41cc service neutron] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:31:04.880165 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1181/2374] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:31:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:31:05.174380 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa1bd55c-44a9-4fef-8276-19023f33a267 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:31:05.175992 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-fa1bd55c-44a9-4fef-8276-19023f33a267 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:31:05.186192 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:INSERT=3,keystone:DELETE=1,keystone:SELECT=96 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:31:05.190172 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa1bd55c-44a9-4fef-8276-19023f33a267 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:31:05.197452 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ea820cb-b732-4a68-a80d-e51ebb0a361f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:31:05.198176 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ea820cb-b732-4a68-a80d-e51ebb0a361f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:31:05.198532 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ea820cb-b732-4a68-a80d-e51ebb0a361f admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:31:05.198818 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ea820cb-b732-4a68-a80d-e51ebb0a361f admin admin] PATH_INFO: `/v3/projects/765f789deef04206a3f89cfda8d1ee08` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:31:05.200621 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ea820cb-b732-4a68-a80d-e51ebb0a361f admin admin] RBAC: Authorizing `identity:delete_project(project_id=765f789deef04206a3f89cfda8d1ee08)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:31:05.203434 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ea820cb-b732-4a68-a80d-e51ebb0a361f admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:31:05.264390 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-9ea820cb-b732-4a68-a80d-e51ebb0a361f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '765f789deef04206a3f89cfda8d1ee08', 'request_id': 'req-9ea820cb-b732-4a68-a80d-e51ebb0a361f'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:31:05.272847 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-9ea820cb-b732-4a68-a80d-e51ebb0a361f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '765f789deef04206a3f89cfda8d1ee08', 'request_id': 'req-9ea820cb-b732-4a68-a80d-e51ebb0a361f'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:31:05.279310 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-9ea820cb-b732-4a68-a80d-e51ebb0a361f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '765f789deef04206a3f89cfda8d1ee08', 'request_id': 'req-9ea820cb-b732-4a68-a80d-e51ebb0a361f'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:31:05.297992 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-9ea820cb-b732-4a68-a80d-e51ebb0a361f admin admin] The token cache is being invalidate because project 765f789deef04206a3f89cfda8d1ee08 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81508) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 01:31:05.298275 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-9ea820cb-b732-4a68-a80d-e51ebb0a361f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:31:05.298898 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-9ea820cb-b732-4a68-a80d-e51ebb0a361f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:31:05.300403 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1194/2375] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 01:31:05 2022] DELETE /identity/v3/projects/765f789deef04206a3f89cfda8d1ee08 => generated 0 bytes in 127 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:31:05.308879 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f219d40d-2901-4a98-b348-6b7c2d9d1d4d None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:31:05.309503 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f219d40d-2901-4a98-b348-6b7c2d9d1d4d None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:31:05.310089 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f219d40d-2901-4a98-b348-6b7c2d9d1d4d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:31:05.332058 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-f219d40d-2901-4a98-b348-6b7c2d9d1d4d None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:31:05.333288 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-f219d40d-2901-4a98-b348-6b7c2d9d1d4d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:31:05.406094 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1182/2376] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:31:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 28 01:31:05.475894 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-add4daee-6737-4a6a-9697-8f2b4c793cc9 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:31:05.476120 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-add4daee-6737-4a6a-9697-8f2b4c793cc9 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:31:05.476352 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-add4daee-6737-4a6a-9697-8f2b4c793cc9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:31:05.501998 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-add4daee-6737-4a6a-9697-8f2b4c793cc9 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:31:05.503240 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-add4daee-6737-4a6a-9697-8f2b4c793cc9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:31:05.512515 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1195/2377] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:31:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:31:05.516795 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f219d40d-2901-4a98-b348-6b7c2d9d1d4d None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:31:05.527715 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f219d40d-2901-4a98-b348-6b7c2d9d1d4d None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:31:05.532617 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-018d8d5b-8fbc-419b-840b-b1edb65fa177 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:31:05.533365 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-018d8d5b-8fbc-419b-840b-b1edb65fa177 admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:31:05.533640 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-018d8d5b-8fbc-419b-840b-b1edb65fa177 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:31:05.533905 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-018d8d5b-8fbc-419b-840b-b1edb65fa177 admin admin] PATH_INFO: `/v3/domains` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:31:05.534547 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-018d8d5b-8fbc-419b-840b-b1edb65fa177 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 01:31:05.534842 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-018d8d5b-8fbc-419b-840b-b1edb65fa177 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:31:05.536444 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-018d8d5b-8fbc-419b-840b-b1edb65fa177 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:31:05.548381 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1183/2378] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Aug 28 01:31:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 01:31:05.552883 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-add4daee-6737-4a6a-9697-8f2b4c793cc9 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:31:05.562998 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-add4daee-6737-4a6a-9697-8f2b4c793cc9 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:31:05.567845 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f676093-46a5-4a14-aa5c-c1115eba4750 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:31:05.568440 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f676093-46a5-4a14-aa5c-c1115eba4750 admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:31:05.568712 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f676093-46a5-4a14-aa5c-c1115eba4750 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:31:05.569008 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f676093-46a5-4a14-aa5c-c1115eba4750 admin admin] PATH_INFO: `/v3/projects` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:31:05.569647 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f676093-46a5-4a14-aa5c-c1115eba4750 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:31:05.571557 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f676093-46a5-4a14-aa5c-c1115eba4750 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:31:05.602558 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1196/2379] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 01:31:05 2022] POST /identity/v3/projects => generated 367 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 28 01:31:05.606814 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-018d8d5b-8fbc-419b-840b-b1edb65fa177 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:31:05.617050 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-018d8d5b-8fbc-419b-840b-b1edb65fa177 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:31:05.645242 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=93,keystone:DELETE=6,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:31:05.676120 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14292589-5ca6-4431-891b-0056f7f66557 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:31:05.676729 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14292589-5ca6-4431-891b-0056f7f66557 admin admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:31:05.677041 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14292589-5ca6-4431-891b-0056f7f66557 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:31:05.677406 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14292589-5ca6-4431-891b-0056f7f66557 admin admin] PATH_INFO: `/v3/users` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:31:05.678037 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14292589-5ca6-4431-891b-0056f7f66557 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:31:05.679962 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14292589-5ca6-4431-891b-0056f7f66557 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:31:05.703132 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1184/2380] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 01:31:05 2022] POST /identity/v3/users => generated 284 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 01:31:05.708012 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f676093-46a5-4a14-aa5c-c1115eba4750 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:31:05.723234 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f676093-46a5-4a14-aa5c-c1115eba4750 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:31:05.729352 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-386ad6df-c1ae-47fa-9d96-3695b8701e21 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:31:05.729919 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-386ad6df-c1ae-47fa-9d96-3695b8701e21 admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:31:05.730206 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-386ad6df-c1ae-47fa-9d96-3695b8701e21 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:31:05.730480 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-386ad6df-c1ae-47fa-9d96-3695b8701e21 admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:31:05.731295 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-386ad6df-c1ae-47fa-9d96-3695b8701e21 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:31:05.732748 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-386ad6df-c1ae-47fa-9d96-3695b8701e21 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:31:05.743547 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1197/2381] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 01:31:05 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 01:31:05.748126 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14292589-5ca6-4431-891b-0056f7f66557 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:31:05.759193 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14292589-5ca6-4431-891b-0056f7f66557 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:31:05.764733 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16222076-0965-445b-81db-98c7815e1b76 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:31:05.765385 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16222076-0965-445b-81db-98c7815e1b76 admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:31:05.765665 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16222076-0965-445b-81db-98c7815e1b76 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:31:05.765950 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16222076-0965-445b-81db-98c7815e1b76 admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:31:05.766604 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16222076-0965-445b-81db-98c7815e1b76 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:31:05.768286 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16222076-0965-445b-81db-98c7815e1b76 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:31:05.779620 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1185/2382] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 01:31:05 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 01:31:05.783956 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-386ad6df-c1ae-47fa-9d96-3695b8701e21 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:31:05.794140 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-386ad6df-c1ae-47fa-9d96-3695b8701e21 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:31:05.798727 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea381a9d-2a60-4b10-a1a9-023f6f899f36 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:31:05.799374 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea381a9d-2a60-4b10-a1a9-023f6f899f36 admin admin] REQUEST_METHOD: `PUT` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:31:05.799651 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea381a9d-2a60-4b10-a1a9-023f6f899f36 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:31:05.799921 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea381a9d-2a60-4b10-a1a9-023f6f899f36 admin admin] PATH_INFO: `/v3/projects/cc620fdae8e8433b8eeb8ae837068a15/users/5bd8d215bb1f4460824df3dc453198fc/roles/0487ef7e943e4701915cb8ce0f2899d7` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:31:05.815092 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea381a9d-2a60-4b10-a1a9-023f6f899f36 admin admin] RBAC: Authorizing `identity:create_grant(project_id=cc620fdae8e8433b8eeb8ae837068a15, user_id=5bd8d215bb1f4460824df3dc453198fc, role_id=0487ef7e943e4701915cb8ce0f2899d7)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:31:05.817822 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea381a9d-2a60-4b10-a1a9-023f6f899f36 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:31:05.833836 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1198/2383] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 01:31:05 2022] PUT /identity/v3/projects/cc620fdae8e8433b8eeb8ae837068a15/users/5bd8d215bb1f4460824df3dc453198fc/roles/0487ef7e943e4701915cb8ce0f2899d7 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:31:05.843926 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c0296bd-c443-4420-bf2f-36ef0dfb2583 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:31:05.844256 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c0296bd-c443-4420-bf2f-36ef0dfb2583 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:31:05.844480 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c0296bd-c443-4420-bf2f-36ef0dfb2583 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:31:05.873592 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-5c0296bd-c443-4420-bf2f-36ef0dfb2583 None None] MFA Rules not processed for user `5bd8d215bb1f4460824df3dc453198fc`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:31:05.874833 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-5c0296bd-c443-4420-bf2f-36ef0dfb2583 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:31:05.942384 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1186/2384] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:31:05 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:31:05.947571 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea381a9d-2a60-4b10-a1a9-023f6f899f36 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:31:06.016264 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea381a9d-2a60-4b10-a1a9-023f6f899f36 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:31:06.021438 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6902720-211d-4135-8505-eb540db27a62 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:31:06.022037 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6902720-211d-4135-8505-eb540db27a62 admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:31:06.022308 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6902720-211d-4135-8505-eb540db27a62 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:31:06.022579 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6902720-211d-4135-8505-eb540db27a62 admin admin] PATH_INFO: `/v3/projects` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:31:06.023201 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6902720-211d-4135-8505-eb540db27a62 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:31:06.025132 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6902720-211d-4135-8505-eb540db27a62 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:31:06.053812 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1199/2385] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 01:31:05 2022] POST /identity/v3/projects => generated 367 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 01:31:06.058620 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c0296bd-c443-4420-bf2f-36ef0dfb2583 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:31:06.071624 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c0296bd-c443-4420-bf2f-36ef0dfb2583 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:31:06.135079 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dc18fb5-da12-4f30-b405-a7c3b58a8a59 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:31:06.135699 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc18fb5-da12-4f30-b405-a7c3b58a8a59 admin admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:31:06.135976 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc18fb5-da12-4f30-b405-a7c3b58a8a59 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:31:06.136246 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc18fb5-da12-4f30-b405-a7c3b58a8a59 admin admin] PATH_INFO: `/v3/users` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:31:06.136881 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dc18fb5-da12-4f30-b405-a7c3b58a8a59 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:31:06.138867 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dc18fb5-da12-4f30-b405-a7c3b58a8a59 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:31:06.163995 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1187/2386] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 01:31:06 2022] POST /identity/v3/users => generated 290 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 01:31:06.168243 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6902720-211d-4135-8505-eb540db27a62 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:31:06.179427 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6902720-211d-4135-8505-eb540db27a62 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:31:06.185237 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea33b397-07e2-4bdb-893d-bc8e0c3178a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:31:06.185834 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea33b397-07e2-4bdb-893d-bc8e0c3178a1 admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:31:06.186115 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea33b397-07e2-4bdb-893d-bc8e0c3178a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:31:06.186378 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea33b397-07e2-4bdb-893d-bc8e0c3178a1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:31:06.187014 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea33b397-07e2-4bdb-893d-bc8e0c3178a1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:31:06.188617 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea33b397-07e2-4bdb-893d-bc8e0c3178a1 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:31:06.191013 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:31:06.202377 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1200/2387] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 01:31:06 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 28 01:31:06.207110 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dc18fb5-da12-4f30-b405-a7c3b58a8a59 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:31:06.218076 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dc18fb5-da12-4f30-b405-a7c3b58a8a59 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:31:06.223382 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3907aa9-c460-41c7-8cec-62b2861c9493 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:31:06.224061 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3907aa9-c460-41c7-8cec-62b2861c9493 admin admin] REQUEST_METHOD: `PUT` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:31:06.224334 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3907aa9-c460-41c7-8cec-62b2861c9493 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:31:06.224615 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3907aa9-c460-41c7-8cec-62b2861c9493 admin admin] PATH_INFO: `/v3/projects/fa385c6c5cb74bbdb1cf493a534c0688/users/260a23e5a7b74f69b8c18376af6feae4/roles/f08d6043261747e3b4763c744eef7dfb` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:31:06.240671 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3907aa9-c460-41c7-8cec-62b2861c9493 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fa385c6c5cb74bbdb1cf493a534c0688, user_id=260a23e5a7b74f69b8c18376af6feae4, role_id=f08d6043261747e3b4763c744eef7dfb)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:31:06.243367 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3907aa9-c460-41c7-8cec-62b2861c9493 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:31:06.261512 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1188/2388] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 01:31:06 2022] PUT /identity/v3/projects/fa385c6c5cb74bbdb1cf493a534c0688/users/260a23e5a7b74f69b8c18376af6feae4/roles/f08d6043261747e3b4763c744eef7dfb => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:31:06.267379 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea33b397-07e2-4bdb-893d-bc8e0c3178a1 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:31:06.281962 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea33b397-07e2-4bdb-893d-bc8e0c3178a1 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:31:06.350729 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1a3e388-f4ef-4114-a7fb-735eb9398774 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:31:06.351338 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a3e388-f4ef-4114-a7fb-735eb9398774 admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:31:06.351708 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a3e388-f4ef-4114-a7fb-735eb9398774 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:31:06.351982 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a3e388-f4ef-4114-a7fb-735eb9398774 admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:31:06.352617 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1a3e388-f4ef-4114-a7fb-735eb9398774 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:31:06.354285 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1a3e388-f4ef-4114-a7fb-735eb9398774 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:31:06.364378 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1201/2389] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 01:31:06 2022] GET /identity/v3/roles => generated 1420 bytes in 98 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 28 01:31:06.369040 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3907aa9-c460-41c7-8cec-62b2861c9493 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:31:06.375984 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:31:06.380120 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3907aa9-c460-41c7-8cec-62b2861c9493 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:31:06.386721 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75706d87-d20a-4ceb-8677-dd0f49fc2414 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:31:06.387395 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75706d87-d20a-4ceb-8677-dd0f49fc2414 admin admin] REQUEST_METHOD: `PUT` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:31:06.387864 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75706d87-d20a-4ceb-8677-dd0f49fc2414 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:31:06.388139 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75706d87-d20a-4ceb-8677-dd0f49fc2414 admin admin] PATH_INFO: `/v3/projects/fa385c6c5cb74bbdb1cf493a534c0688/users/260a23e5a7b74f69b8c18376af6feae4/roles/f08d6043261747e3b4763c744eef7dfb` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:31:06.390099 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75706d87-d20a-4ceb-8677-dd0f49fc2414 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fa385c6c5cb74bbdb1cf493a534c0688, user_id=260a23e5a7b74f69b8c18376af6feae4, role_id=f08d6043261747e3b4763c744eef7dfb)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:31:06.392747 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75706d87-d20a-4ceb-8677-dd0f49fc2414 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:31:06.403093 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1189/2390] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 01:31:06 2022] PUT /identity/v3/projects/fa385c6c5cb74bbdb1cf493a534c0688/users/260a23e5a7b74f69b8c18376af6feae4/roles/f08d6043261747e3b4763c744eef7dfb => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:31:06.415042 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b52e25f2-1f16-4197-a542-7085825b602b None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:31:06.415314 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b52e25f2-1f16-4197-a542-7085825b602b None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:31:06.415601 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b52e25f2-1f16-4197-a542-7085825b602b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:31:06.445406 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-b52e25f2-1f16-4197-a542-7085825b602b None None] MFA Rules not processed for user `260a23e5a7b74f69b8c18376af6feae4`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:31:06.446632 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-b52e25f2-1f16-4197-a542-7085825b602b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:31:06.521711 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1202/2391] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:31:06 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:31:06.533176 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75706d87-d20a-4ceb-8677-dd0f49fc2414 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:31:06.535431 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-75706d87-d20a-4ceb-8677-dd0f49fc2414 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:31:06.607994 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75706d87-d20a-4ceb-8677-dd0f49fc2414 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:31:06.612244 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77b7e9e7-2818-4480-8f8a-edd40f11f07e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:31:06.612885 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b7e9e7-2818-4480-8f8a-edd40f11f07e service cinder] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:31:06.613189 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b7e9e7-2818-4480-8f8a-edd40f11f07e service cinder] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:31:06.613462 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b7e9e7-2818-4480-8f8a-edd40f11f07e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:31:06.623710 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77b7e9e7-2818-4480-8f8a-edd40f11f07e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:31:06.625184 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77b7e9e7-2818-4480-8f8a-edd40f11f07e service cinder] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:31:06.635767 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1190/2392] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:31:06 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:31:06.727895 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b52e25f2-1f16-4197-a542-7085825b602b None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:31:06.738581 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b52e25f2-1f16-4197-a542-7085825b602b None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:31:06.742901 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eafc359-9140-49f4-8cd1-a9b34ea3cdc4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:31:06.743544 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eafc359-9140-49f4-8cd1-a9b34ea3cdc4 service cinder] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:31:06.743819 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eafc359-9140-49f4-8cd1-a9b34ea3cdc4 service cinder] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:31:06.744113 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eafc359-9140-49f4-8cd1-a9b34ea3cdc4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:31:06.807098 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eafc359-9140-49f4-8cd1-a9b34ea3cdc4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:31:06.808596 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eafc359-9140-49f4-8cd1-a9b34ea3cdc4 service cinder] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:31:06.818314 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1203/2393] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:31:06 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:31:11.640758 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77b7e9e7-2818-4480-8f8a-edd40f11f07e service cinder] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:31:11.710904 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77b7e9e7-2818-4480-8f8a-edd40f11f07e service cinder] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:31:11.715557 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90fb475b-e505-42e5-a7e9-a717d8c8c8b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:31:11.716118 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90fb475b-e505-42e5-a7e9-a717d8c8c8b7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:31:11.716391 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90fb475b-e505-42e5-a7e9-a717d8c8c8b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:31:11.716657 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90fb475b-e505-42e5-a7e9-a717d8c8c8b7 admin admin] PATH_INFO: `/v3/users/5bd8d215bb1f4460824df3dc453198fc` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:31:11.717811 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90fb475b-e505-42e5-a7e9-a717d8c8c8b7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5bd8d215bb1f4460824df3dc453198fc)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:31:11.719753 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90fb475b-e505-42e5-a7e9-a717d8c8c8b7 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:31:11.770405 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-90fb475b-e505-42e5-a7e9-a717d8c8c8b7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5bd8d215bb1f4460824df3dc453198fc'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:31:11.782173 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-90fb475b-e505-42e5-a7e9-a717d8c8c8b7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5bd8d215bb1f4460824df3dc453198fc'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:31:11.790565 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-90fb475b-e505-42e5-a7e9-a717d8c8c8b7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5bd8d215bb1f4460824df3dc453198fc'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:31:11.797762 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-90fb475b-e505-42e5-a7e9-a717d8c8c8b7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5bd8d215bb1f4460824df3dc453198fc'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:31:11.798416 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-90fb475b-e505-42e5-a7e9-a717d8c8c8b7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5bd8d215bb1f4460824df3dc453198fc'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:31:11.804730 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1191/2394] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 01:31:11 2022] DELETE /identity/v3/users/5bd8d215bb1f4460824df3dc453198fc => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:31:11.809401 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f86dde5-c7c2-44f7-aa2a-2b205c93930f None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:31:11.810241 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f86dde5-c7c2-44f7-aa2a-2b205c93930f None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:31:11.810371 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f86dde5-c7c2-44f7-aa2a-2b205c93930f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:31:11.830524 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-3f86dde5-c7c2-44f7-aa2a-2b205c93930f None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:31:11.831733 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-3f86dde5-c7c2-44f7-aa2a-2b205c93930f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:31:11.845242 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:31:11.895230 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1204/2395] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:31:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:31:11.905651 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90fb475b-e505-42e5-a7e9-a717d8c8c8b7 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:31:11.907304 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-90fb475b-e505-42e5-a7e9-a717d8c8c8b7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:31:11.922966 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:31:11.968401 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90fb475b-e505-42e5-a7e9-a717d8c8c8b7 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:31:11.972066 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00c68c7c-37d8-440f-9c27-48db71b25558 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:31:11.972670 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00c68c7c-37d8-440f-9c27-48db71b25558 service neutron] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:31:11.972943 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00c68c7c-37d8-440f-9c27-48db71b25558 service neutron] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:31:11.973243 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00c68c7c-37d8-440f-9c27-48db71b25558 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:31:11.982527 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00c68c7c-37d8-440f-9c27-48db71b25558 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:31:11.983954 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00c68c7c-37d8-440f-9c27-48db71b25558 service neutron] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:31:11.993860 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1192/2396] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:31:11 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:31:12.270285 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f86dde5-c7c2-44f7-aa2a-2b205c93930f None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:31:12.271749 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-3f86dde5-c7c2-44f7-aa2a-2b205c93930f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:31:12.284387 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f86dde5-c7c2-44f7-aa2a-2b205c93930f None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:31:12.289797 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec23d971-8a9d-49f7-a974-6c37d23a4e80 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:31:12.290245 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec23d971-8a9d-49f7-a974-6c37d23a4e80 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:31:12.290528 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec23d971-8a9d-49f7-a974-6c37d23a4e80 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:31:12.290799 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec23d971-8a9d-49f7-a974-6c37d23a4e80 admin admin] PATH_INFO: `/v3/projects/cc620fdae8e8433b8eeb8ae837068a15` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:31:12.291967 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec23d971-8a9d-49f7-a974-6c37d23a4e80 admin admin] RBAC: Authorizing `identity:delete_project(project_id=cc620fdae8e8433b8eeb8ae837068a15)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:31:12.293922 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec23d971-8a9d-49f7-a974-6c37d23a4e80 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:31:12.359478 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-ec23d971-8a9d-49f7-a974-6c37d23a4e80 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cc620fdae8e8433b8eeb8ae837068a15', 'request_id': 'req-ec23d971-8a9d-49f7-a974-6c37d23a4e80'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:31:12.365662 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-ec23d971-8a9d-49f7-a974-6c37d23a4e80 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cc620fdae8e8433b8eeb8ae837068a15', 'request_id': 'req-ec23d971-8a9d-49f7-a974-6c37d23a4e80'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:31:12.372511 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-ec23d971-8a9d-49f7-a974-6c37d23a4e80 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cc620fdae8e8433b8eeb8ae837068a15', 'request_id': 'req-ec23d971-8a9d-49f7-a974-6c37d23a4e80'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:31:12.387226 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-ec23d971-8a9d-49f7-a974-6c37d23a4e80 admin admin] The token cache is being invalidate because project cc620fdae8e8433b8eeb8ae837068a15 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81508) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 01:31:12.387514 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-ec23d971-8a9d-49f7-a974-6c37d23a4e80 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:31:12.388144 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-ec23d971-8a9d-49f7-a974-6c37d23a4e80 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:31:12.389699 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1205/2397] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 01:31:12 2022] DELETE /identity/v3/projects/cc620fdae8e8433b8eeb8ae837068a15 => generated 0 bytes in 120 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:31:12.393817 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00c68c7c-37d8-440f-9c27-48db71b25558 service neutron] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:31:12.403809 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00c68c7c-37d8-440f-9c27-48db71b25558 service neutron] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:31:12.466780 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac4304e8-dc69-4431-98a1-c47d187a5d7e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:31:12.467412 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac4304e8-dc69-4431-98a1-c47d187a5d7e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:31:12.467692 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac4304e8-dc69-4431-98a1-c47d187a5d7e admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:31:12.467961 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac4304e8-dc69-4431-98a1-c47d187a5d7e admin admin] PATH_INFO: `/v3/users/260a23e5a7b74f69b8c18376af6feae4` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:31:12.469123 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac4304e8-dc69-4431-98a1-c47d187a5d7e admin admin] RBAC: Authorizing `identity:delete_user(user_id=260a23e5a7b74f69b8c18376af6feae4)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:31:12.471017 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac4304e8-dc69-4431-98a1-c47d187a5d7e admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:31:12.519021 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-ac4304e8-dc69-4431-98a1-c47d187a5d7e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '260a23e5a7b74f69b8c18376af6feae4'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:31:12.531850 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-ac4304e8-dc69-4431-98a1-c47d187a5d7e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '260a23e5a7b74f69b8c18376af6feae4'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:31:12.541147 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-ac4304e8-dc69-4431-98a1-c47d187a5d7e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '260a23e5a7b74f69b8c18376af6feae4'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:31:12.548792 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-ac4304e8-dc69-4431-98a1-c47d187a5d7e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '260a23e5a7b74f69b8c18376af6feae4'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:31:12.549509 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-ac4304e8-dc69-4431-98a1-c47d187a5d7e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '260a23e5a7b74f69b8c18376af6feae4'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:31:12.556723 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1193/2398] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 01:31:12 2022] DELETE /identity/v3/users/260a23e5a7b74f69b8c18376af6feae4 => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:31:12.853321 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec23d971-8a9d-49f7-a974-6c37d23a4e80 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:31:12.854831 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-ec23d971-8a9d-49f7-a974-6c37d23a4e80 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:31:12.868671 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec23d971-8a9d-49f7-a974-6c37d23a4e80 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:31:12.926193 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa90424a-57da-42e9-983b-4a6302757c7a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:31:12.926753 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa90424a-57da-42e9-983b-4a6302757c7a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:31:12.927031 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa90424a-57da-42e9-983b-4a6302757c7a admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:31:12.927355 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa90424a-57da-42e9-983b-4a6302757c7a admin admin] PATH_INFO: `/v3/projects/fa385c6c5cb74bbdb1cf493a534c0688` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:31:12.928614 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa90424a-57da-42e9-983b-4a6302757c7a admin admin] RBAC: Authorizing `identity:delete_project(project_id=fa385c6c5cb74bbdb1cf493a534c0688)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:31:12.930592 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa90424a-57da-42e9-983b-4a6302757c7a admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:31:12.979365 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:31:12.983269 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-aa90424a-57da-42e9-983b-4a6302757c7a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fa385c6c5cb74bbdb1cf493a534c0688', 'request_id': 'req-aa90424a-57da-42e9-983b-4a6302757c7a'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:31:12.989437 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-aa90424a-57da-42e9-983b-4a6302757c7a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fa385c6c5cb74bbdb1cf493a534c0688', 'request_id': 'req-aa90424a-57da-42e9-983b-4a6302757c7a'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:31:12.996102 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-aa90424a-57da-42e9-983b-4a6302757c7a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fa385c6c5cb74bbdb1cf493a534c0688', 'request_id': 'req-aa90424a-57da-42e9-983b-4a6302757c7a'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:31:13.011336 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-aa90424a-57da-42e9-983b-4a6302757c7a admin admin] The token cache is being invalidate because project fa385c6c5cb74bbdb1cf493a534c0688 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81508) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 01:31:13.011621 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-aa90424a-57da-42e9-983b-4a6302757c7a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:31:13.012284 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-aa90424a-57da-42e9-983b-4a6302757c7a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:31:13.013871 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1206/2399] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 01:31:12 2022] DELETE /identity/v3/projects/fa385c6c5cb74bbdb1cf493a534c0688 => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:31:13.019909 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18a30e1f-5012-4c0e-bbc9-5bc46ff22ca7 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:31:13.020364 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18a30e1f-5012-4c0e-bbc9-5bc46ff22ca7 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:31:13.020442 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18a30e1f-5012-4c0e-bbc9-5bc46ff22ca7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:31:13.040646 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-18a30e1f-5012-4c0e-bbc9-5bc46ff22ca7 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:31:13.041874 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-18a30e1f-5012-4c0e-bbc9-5bc46ff22ca7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:31:13.072704 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:31:13.107470 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1194/2400] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:31:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:31:13.115480 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03eddcaf-f48e-4568-a15c-b48b4f0fb386 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:31:13.115762 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03eddcaf-f48e-4568-a15c-b48b4f0fb386 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:31:13.116035 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03eddcaf-f48e-4568-a15c-b48b4f0fb386 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:31:13.138575 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-03eddcaf-f48e-4568-a15c-b48b4f0fb386 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:31:13.139784 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-03eddcaf-f48e-4568-a15c-b48b4f0fb386 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:31:13.148722 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1207/2401] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:31:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:31:13.153466 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18a30e1f-5012-4c0e-bbc9-5bc46ff22ca7 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:31:13.163834 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18a30e1f-5012-4c0e-bbc9-5bc46ff22ca7 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:31:13.168931 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27c6e0f0-0ae9-4d23-aceb-080b6560f416 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:31:13.169578 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27c6e0f0-0ae9-4d23-aceb-080b6560f416 admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:31:13.169854 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27c6e0f0-0ae9-4d23-aceb-080b6560f416 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:31:13.170117 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27c6e0f0-0ae9-4d23-aceb-080b6560f416 admin admin] PATH_INFO: `/v3/domains` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:31:13.170790 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27c6e0f0-0ae9-4d23-aceb-080b6560f416 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 01:31:13.171088 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27c6e0f0-0ae9-4d23-aceb-080b6560f416 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:31:13.172683 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27c6e0f0-0ae9-4d23-aceb-080b6560f416 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:31:13.185013 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1195/2402] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Aug 28 01:31:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 01:31:13.189469 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03eddcaf-f48e-4568-a15c-b48b4f0fb386 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:31:13.198860 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03eddcaf-f48e-4568-a15c-b48b4f0fb386 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:31:13.204540 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abf8e92b-af84-49c2-9921-f25be9399cf6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:31:13.205162 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abf8e92b-af84-49c2-9921-f25be9399cf6 admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:31:13.205436 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abf8e92b-af84-49c2-9921-f25be9399cf6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:31:13.205697 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abf8e92b-af84-49c2-9921-f25be9399cf6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:31:13.206335 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abf8e92b-af84-49c2-9921-f25be9399cf6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:31:13.208238 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abf8e92b-af84-49c2-9921-f25be9399cf6 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:31:13.240000 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1208/2403] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 01:31:13 2022] POST /identity/v3/projects => generated 373 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 28 01:31:13.244609 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27c6e0f0-0ae9-4d23-aceb-080b6560f416 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:31:13.255177 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27c6e0f0-0ae9-4d23-aceb-080b6560f416 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:31:13.314466 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d40356c-9053-491d-88f7-7d9241b29b90 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:31:13.315105 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d40356c-9053-491d-88f7-7d9241b29b90 admin admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:31:13.315385 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d40356c-9053-491d-88f7-7d9241b29b90 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:31:13.315655 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d40356c-9053-491d-88f7-7d9241b29b90 admin admin] PATH_INFO: `/v3/users` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:31:13.316293 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d40356c-9053-491d-88f7-7d9241b29b90 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:31:13.318265 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d40356c-9053-491d-88f7-7d9241b29b90 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:31:13.345203 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1196/2404] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 01:31:13 2022] POST /identity/v3/users => generated 287 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 28 01:31:13.349358 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abf8e92b-af84-49c2-9921-f25be9399cf6 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:31:13.360029 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abf8e92b-af84-49c2-9921-f25be9399cf6 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:31:13.365776 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f66506e1-29e0-4e4e-ac07-ad906b0246f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:31:13.366463 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f66506e1-29e0-4e4e-ac07-ad906b0246f6 admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:31:13.366901 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f66506e1-29e0-4e4e-ac07-ad906b0246f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:31:13.367131 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f66506e1-29e0-4e4e-ac07-ad906b0246f6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:31:13.367870 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f66506e1-29e0-4e4e-ac07-ad906b0246f6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:31:13.369685 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f66506e1-29e0-4e4e-ac07-ad906b0246f6 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:31:13.380294 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1209/2405] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 01:31:13 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 01:31:13.384648 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d40356c-9053-491d-88f7-7d9241b29b90 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:31:13.396469 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d40356c-9053-491d-88f7-7d9241b29b90 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:31:13.403635 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d0ccb6a-45df-4fb7-a89c-5cbd4c1a13bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:31:13.405099 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d0ccb6a-45df-4fb7-a89c-5cbd4c1a13bc admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:31:13.405099 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d0ccb6a-45df-4fb7-a89c-5cbd4c1a13bc admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:31:13.405099 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d0ccb6a-45df-4fb7-a89c-5cbd4c1a13bc admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:31:13.405685 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d0ccb6a-45df-4fb7-a89c-5cbd4c1a13bc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:31:13.407687 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d0ccb6a-45df-4fb7-a89c-5cbd4c1a13bc admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:31:13.423817 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1197/2406] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 01:31:13 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 01:31:13.428241 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f66506e1-29e0-4e4e-ac07-ad906b0246f6 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:31:13.439285 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f66506e1-29e0-4e4e-ac07-ad906b0246f6 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:31:13.445281 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c6a8c29-1b8a-4c55-8a9f-86e1d44b66f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:31:13.445910 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c6a8c29-1b8a-4c55-8a9f-86e1d44b66f0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:31:13.446183 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c6a8c29-1b8a-4c55-8a9f-86e1d44b66f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:31:13.446447 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c6a8c29-1b8a-4c55-8a9f-86e1d44b66f0 admin admin] PATH_INFO: `/v3/projects/be106c6b074341768bfa7fd3732fc12e/users/0bbf188ec2ee409da469230ece05ea52/roles/0487ef7e943e4701915cb8ce0f2899d7` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:31:13.463372 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c6a8c29-1b8a-4c55-8a9f-86e1d44b66f0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=be106c6b074341768bfa7fd3732fc12e, user_id=0bbf188ec2ee409da469230ece05ea52, role_id=0487ef7e943e4701915cb8ce0f2899d7)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:31:13.466038 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c6a8c29-1b8a-4c55-8a9f-86e1d44b66f0 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:31:13.485332 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1210/2407] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 01:31:13 2022] PUT /identity/v3/projects/be106c6b074341768bfa7fd3732fc12e/users/0bbf188ec2ee409da469230ece05ea52/roles/0487ef7e943e4701915cb8ce0f2899d7 => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 28 01:31:13.501243 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33f9316b-8563-43ff-877c-188a5219a59e None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:31:13.501716 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33f9316b-8563-43ff-877c-188a5219a59e None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:31:13.502180 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33f9316b-8563-43ff-877c-188a5219a59e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:31:13.531857 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-33f9316b-8563-43ff-877c-188a5219a59e None None] MFA Rules not processed for user `0bbf188ec2ee409da469230ece05ea52`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:31:13.533102 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-33f9316b-8563-43ff-877c-188a5219a59e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:31:13.605982 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1198/2408] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:31:13 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:31:13.617427 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c6a8c29-1b8a-4c55-8a9f-86e1d44b66f0 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:31:13.619190 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-2c6a8c29-1b8a-4c55-8a9f-86e1d44b66f0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:31:13.677391 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c6a8c29-1b8a-4c55-8a9f-86e1d44b66f0 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:31:13.681496 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cd6910d-5c84-4aa7-8012-0a88ca9a4e59 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:31:13.682107 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cd6910d-5c84-4aa7-8012-0a88ca9a4e59 service cinder] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:31:13.682385 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cd6910d-5c84-4aa7-8012-0a88ca9a4e59 service cinder] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:31:13.682657 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cd6910d-5c84-4aa7-8012-0a88ca9a4e59 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:31:13.692356 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cd6910d-5c84-4aa7-8012-0a88ca9a4e59 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:31:13.693804 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cd6910d-5c84-4aa7-8012-0a88ca9a4e59 service cinder] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:31:13.703174 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1211/2409] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:31:13 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:31:13.726182 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33f9316b-8563-43ff-877c-188a5219a59e None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:31:13.761115 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:31:13.790916 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33f9316b-8563-43ff-877c-188a5219a59e None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:31:13.795908 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c48f3298-37a3-401a-9a54-687cc6933e6a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:31:13.796551 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c48f3298-37a3-401a-9a54-687cc6933e6a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:31:13.796828 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c48f3298-37a3-401a-9a54-687cc6933e6a admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:31:13.797129 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c48f3298-37a3-401a-9a54-687cc6933e6a admin admin] PATH_INFO: `/v3/users/0bbf188ec2ee409da469230ece05ea52` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:31:13.798296 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c48f3298-37a3-401a-9a54-687cc6933e6a admin admin] RBAC: Authorizing `identity:delete_user(user_id=0bbf188ec2ee409da469230ece05ea52)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:31:13.800282 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c48f3298-37a3-401a-9a54-687cc6933e6a admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:31:13.855256 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-c48f3298-37a3-401a-9a54-687cc6933e6a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0bbf188ec2ee409da469230ece05ea52'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:31:13.869299 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-c48f3298-37a3-401a-9a54-687cc6933e6a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0bbf188ec2ee409da469230ece05ea52'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:31:13.879173 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-c48f3298-37a3-401a-9a54-687cc6933e6a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0bbf188ec2ee409da469230ece05ea52'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:31:13.886544 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-c48f3298-37a3-401a-9a54-687cc6933e6a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0bbf188ec2ee409da469230ece05ea52'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:31:13.887248 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-c48f3298-37a3-401a-9a54-687cc6933e6a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0bbf188ec2ee409da469230ece05ea52'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:31:13.893208 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1199/2410] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 01:31:13 2022] DELETE /identity/v3/users/0bbf188ec2ee409da469230ece05ea52 => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:31:13.897927 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5689364-6fc8-4f48-8fc0-cbd4e648a14f None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:31:13.898394 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5689364-6fc8-4f48-8fc0-cbd4e648a14f None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:31:13.898591 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5689364-6fc8-4f48-8fc0-cbd4e648a14f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:31:13.919211 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-b5689364-6fc8-4f48-8fc0-cbd4e648a14f None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:31:13.920430 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-b5689364-6fc8-4f48-8fc0-cbd4e648a14f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:31:13.986396 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1212/2411] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:31:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:31:13.996382 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c48f3298-37a3-401a-9a54-687cc6933e6a admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:31:13.997970 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-c48f3298-37a3-401a-9a54-687cc6933e6a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:31:14.076142 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c48f3298-37a3-401a-9a54-687cc6933e6a admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:31:14.080210 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb3a0099-408e-44ed-b1dc-2a86fb0946a0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:31:14.080814 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb3a0099-408e-44ed-b1dc-2a86fb0946a0 service neutron] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:31:14.081139 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb3a0099-408e-44ed-b1dc-2a86fb0946a0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:31:14.081400 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb3a0099-408e-44ed-b1dc-2a86fb0946a0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:31:14.090833 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb3a0099-408e-44ed-b1dc-2a86fb0946a0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:31:14.092298 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb3a0099-408e-44ed-b1dc-2a86fb0946a0 service neutron] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:31:14.105582 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1200/2412] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:31:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 110 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:31:14.396997 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5689364-6fc8-4f48-8fc0-cbd4e648a14f None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:31:14.398827 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-b5689364-6fc8-4f48-8fc0-cbd4e648a14f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:31:14.420164 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5689364-6fc8-4f48-8fc0-cbd4e648a14f None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:31:14.425929 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0390f40d-594d-4f74-8af2-7e0b0ec5b729 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:31:14.426614 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0390f40d-594d-4f74-8af2-7e0b0ec5b729 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:31:14.426894 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0390f40d-594d-4f74-8af2-7e0b0ec5b729 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:31:14.427171 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0390f40d-594d-4f74-8af2-7e0b0ec5b729 admin admin] PATH_INFO: `/v3/projects/be106c6b074341768bfa7fd3732fc12e` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:31:14.428494 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0390f40d-594d-4f74-8af2-7e0b0ec5b729 admin admin] RBAC: Authorizing `identity:delete_project(project_id=be106c6b074341768bfa7fd3732fc12e)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:31:14.430606 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0390f40d-594d-4f74-8af2-7e0b0ec5b729 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:31:14.437866 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=93,keystone:DELETE=4,keystone:INSERT=3 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:31:14.495416 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-0390f40d-594d-4f74-8af2-7e0b0ec5b729 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'be106c6b074341768bfa7fd3732fc12e', 'request_id': 'req-0390f40d-594d-4f74-8af2-7e0b0ec5b729'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:31:14.501300 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-0390f40d-594d-4f74-8af2-7e0b0ec5b729 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'be106c6b074341768bfa7fd3732fc12e', 'request_id': 'req-0390f40d-594d-4f74-8af2-7e0b0ec5b729'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:31:14.507720 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-0390f40d-594d-4f74-8af2-7e0b0ec5b729 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'be106c6b074341768bfa7fd3732fc12e', 'request_id': 'req-0390f40d-594d-4f74-8af2-7e0b0ec5b729'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:31:14.522361 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-0390f40d-594d-4f74-8af2-7e0b0ec5b729 admin admin] The token cache is being invalidate because project be106c6b074341768bfa7fd3732fc12e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81508) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 01:31:14.522648 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-0390f40d-594d-4f74-8af2-7e0b0ec5b729 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:31:14.523343 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-0390f40d-594d-4f74-8af2-7e0b0ec5b729 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:31:14.524996 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1213/2413] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 01:31:14 2022] DELETE /identity/v3/projects/be106c6b074341768bfa7fd3732fc12e => generated 0 bytes in 129 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:31:14.530154 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58dea39b-3bbd-4ce8-9463-149553c72281 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:31:14.530529 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58dea39b-3bbd-4ce8-9463-149553c72281 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:31:14.530801 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58dea39b-3bbd-4ce8-9463-149553c72281 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:31:14.551163 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-58dea39b-3bbd-4ce8-9463-149553c72281 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:31:14.552372 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-58dea39b-3bbd-4ce8-9463-149553c72281 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:31:14.592418 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:31:14.613873 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1201/2414] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:31:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:31:14.622964 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82504f9c-64d0-4403-8135-fe39c0a832d2 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:31:14.623253 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82504f9c-64d0-4403-8135-fe39c0a832d2 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:31:14.623527 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82504f9c-64d0-4403-8135-fe39c0a832d2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:31:14.644393 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-82504f9c-64d0-4403-8135-fe39c0a832d2 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:31:14.645648 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-82504f9c-64d0-4403-8135-fe39c0a832d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:31:14.655430 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1214/2415] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:31:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:31:14.660041 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58dea39b-3bbd-4ce8-9463-149553c72281 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:31:14.669943 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58dea39b-3bbd-4ce8-9463-149553c72281 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:31:14.675119 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dc3985b-4b23-4318-ac13-1022197a6f7b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:31:14.675748 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dc3985b-4b23-4318-ac13-1022197a6f7b admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:31:14.676026 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dc3985b-4b23-4318-ac13-1022197a6f7b admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:31:14.676298 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dc3985b-4b23-4318-ac13-1022197a6f7b admin admin] PATH_INFO: `/v3/domains` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:31:14.676935 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dc3985b-4b23-4318-ac13-1022197a6f7b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 01:31:14.677267 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dc3985b-4b23-4318-ac13-1022197a6f7b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:31:14.678926 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dc3985b-4b23-4318-ac13-1022197a6f7b admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:31:14.691081 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1202/2416] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Aug 28 01:31:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 01:31:14.695614 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82504f9c-64d0-4403-8135-fe39c0a832d2 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:31:14.705233 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82504f9c-64d0-4403-8135-fe39c0a832d2 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:31:14.710622 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33fd97bb-b11e-4a23-80c3-108b3a3e6799 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:31:14.711232 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33fd97bb-b11e-4a23-80c3-108b3a3e6799 admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:31:14.711499 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33fd97bb-b11e-4a23-80c3-108b3a3e6799 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:31:14.711793 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33fd97bb-b11e-4a23-80c3-108b3a3e6799 admin admin] PATH_INFO: `/v3/projects` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:31:14.712422 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33fd97bb-b11e-4a23-80c3-108b3a3e6799 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:31:14.714360 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33fd97bb-b11e-4a23-80c3-108b3a3e6799 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:31:14.746303 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1215/2417] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 01:31:14 2022] POST /identity/v3/projects => generated 373 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 01:31:14.750588 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dc3985b-4b23-4318-ac13-1022197a6f7b admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:31:14.760903 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dc3985b-4b23-4318-ac13-1022197a6f7b admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:31:14.819114 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baa78759-ba70-4b0f-ae5e-da092d3b82af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:31:14.819729 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa78759-ba70-4b0f-ae5e-da092d3b82af admin admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:31:14.820006 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa78759-ba70-4b0f-ae5e-da092d3b82af admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:31:14.820278 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa78759-ba70-4b0f-ae5e-da092d3b82af admin admin] PATH_INFO: `/v3/users` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:31:14.820942 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baa78759-ba70-4b0f-ae5e-da092d3b82af admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:31:14.822938 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baa78759-ba70-4b0f-ae5e-da092d3b82af admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:31:14.845992 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1203/2418] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 01:31:14 2022] POST /identity/v3/users => generated 287 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 01:31:14.850291 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33fd97bb-b11e-4a23-80c3-108b3a3e6799 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:31:14.860871 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33fd97bb-b11e-4a23-80c3-108b3a3e6799 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:31:14.866533 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9051818-946f-44b6-9891-6e7b16bd7b28 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:31:14.867129 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9051818-946f-44b6-9891-6e7b16bd7b28 admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:31:14.867405 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9051818-946f-44b6-9891-6e7b16bd7b28 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:31:14.867668 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9051818-946f-44b6-9891-6e7b16bd7b28 admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:31:14.868301 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9051818-946f-44b6-9891-6e7b16bd7b28 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:31:14.869978 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9051818-946f-44b6-9891-6e7b16bd7b28 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:31:14.880088 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1216/2419] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 01:31:14 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 01:31:14.884360 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baa78759-ba70-4b0f-ae5e-da092d3b82af admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:31:14.894932 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baa78759-ba70-4b0f-ae5e-da092d3b82af admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:31:14.900495 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2963f057-eb32-44c3-9e2f-ad0002d49ef7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:31:14.901125 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2963f057-eb32-44c3-9e2f-ad0002d49ef7 admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:31:14.901406 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2963f057-eb32-44c3-9e2f-ad0002d49ef7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:31:14.901676 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2963f057-eb32-44c3-9e2f-ad0002d49ef7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:31:14.902312 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2963f057-eb32-44c3-9e2f-ad0002d49ef7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:31:14.903930 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2963f057-eb32-44c3-9e2f-ad0002d49ef7 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:31:14.914875 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1204/2420] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 01:31:14 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 01:31:14.919170 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9051818-946f-44b6-9891-6e7b16bd7b28 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:31:14.929996 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9051818-946f-44b6-9891-6e7b16bd7b28 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:31:14.935992 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31df4d16-f7eb-4688-8718-a990ad5f8199 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:31:14.936702 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31df4d16-f7eb-4688-8718-a990ad5f8199 admin admin] REQUEST_METHOD: `PUT` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:31:14.936995 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31df4d16-f7eb-4688-8718-a990ad5f8199 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:31:14.937284 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31df4d16-f7eb-4688-8718-a990ad5f8199 admin admin] PATH_INFO: `/v3/projects/7fad852d65d54444b17d925bc42969b5/users/4286629c5a3e4a298165f7d756124c54/roles/0487ef7e943e4701915cb8ce0f2899d7` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:31:14.952658 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31df4d16-f7eb-4688-8718-a990ad5f8199 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7fad852d65d54444b17d925bc42969b5, user_id=4286629c5a3e4a298165f7d756124c54, role_id=0487ef7e943e4701915cb8ce0f2899d7)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:31:14.955354 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31df4d16-f7eb-4688-8718-a990ad5f8199 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:31:14.971871 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1217/2421] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 01:31:14 2022] PUT /identity/v3/projects/7fad852d65d54444b17d925bc42969b5/users/4286629c5a3e4a298165f7d756124c54/roles/0487ef7e943e4701915cb8ce0f2899d7 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:31:14.982539 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbee01a1-2462-459f-8ec2-601a38b781d5 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:31:14.982850 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbee01a1-2462-459f-8ec2-601a38b781d5 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:31:14.983122 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbee01a1-2462-459f-8ec2-601a38b781d5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:31:15.011047 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-dbee01a1-2462-459f-8ec2-601a38b781d5 None None] MFA Rules not processed for user `4286629c5a3e4a298165f7d756124c54`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:31:15.012195 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-dbee01a1-2462-459f-8ec2-601a38b781d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:31:15.083655 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1205/2422] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:31:14 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:31:15.095557 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31df4d16-f7eb-4688-8718-a990ad5f8199 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:31:15.097864 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-31df4d16-f7eb-4688-8718-a990ad5f8199 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:31:15.157107 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31df4d16-f7eb-4688-8718-a990ad5f8199 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:31:15.161238 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d4179f6-a977-430a-860f-d9f215578d53 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:31:15.161942 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d4179f6-a977-430a-860f-d9f215578d53 service cinder] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:31:15.162229 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d4179f6-a977-430a-860f-d9f215578d53 service cinder] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:31:15.162502 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d4179f6-a977-430a-860f-d9f215578d53 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:31:15.171202 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d4179f6-a977-430a-860f-d9f215578d53 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:31:15.172676 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d4179f6-a977-430a-860f-d9f215578d53 service cinder] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:31:15.182586 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1218/2423] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:31:15 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:31:18.268469 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbee01a1-2462-459f-8ec2-601a38b781d5 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:31:18.312083 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:31:18.335723 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbee01a1-2462-459f-8ec2-601a38b781d5 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:31:18.341237 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f97a3efe-87ee-45b6-b440-d0adcb0ac148 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:31:18.341887 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f97a3efe-87ee-45b6-b440-d0adcb0ac148 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:31:18.342160 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f97a3efe-87ee-45b6-b440-d0adcb0ac148 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:31:18.342420 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f97a3efe-87ee-45b6-b440-d0adcb0ac148 admin admin] PATH_INFO: `/v3/users/4286629c5a3e4a298165f7d756124c54` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:31:18.343588 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f97a3efe-87ee-45b6-b440-d0adcb0ac148 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4286629c5a3e4a298165f7d756124c54)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:31:18.345562 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f97a3efe-87ee-45b6-b440-d0adcb0ac148 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:31:18.411482 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-f97a3efe-87ee-45b6-b440-d0adcb0ac148 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4286629c5a3e4a298165f7d756124c54'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:31:18.424689 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-f97a3efe-87ee-45b6-b440-d0adcb0ac148 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4286629c5a3e4a298165f7d756124c54'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:31:18.435589 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-f97a3efe-87ee-45b6-b440-d0adcb0ac148 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4286629c5a3e4a298165f7d756124c54'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:31:18.443622 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-f97a3efe-87ee-45b6-b440-d0adcb0ac148 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4286629c5a3e4a298165f7d756124c54'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:31:18.444331 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-f97a3efe-87ee-45b6-b440-d0adcb0ac148 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4286629c5a3e4a298165f7d756124c54'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:31:18.450671 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1206/2424] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 01:31:18 2022] DELETE /identity/v3/users/4286629c5a3e4a298165f7d756124c54 => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:31:18.455345 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c382b4-6e5c-452f-a700-68b10de4c130 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:31:18.455827 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c382b4-6e5c-452f-a700-68b10de4c130 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:31:18.455911 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c382b4-6e5c-452f-a700-68b10de4c130 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:31:18.475384 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-21c382b4-6e5c-452f-a700-68b10de4c130 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:31:18.476638 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-21c382b4-6e5c-452f-a700-68b10de4c130 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:31:18.508438 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:31:18.539760 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1219/2425] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:31:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:31:18.549943 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f97a3efe-87ee-45b6-b440-d0adcb0ac148 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:31:18.551503 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-f97a3efe-87ee-45b6-b440-d0adcb0ac148 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:31:18.610478 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f97a3efe-87ee-45b6-b440-d0adcb0ac148 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:31:18.614261 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ad74750-db79-482c-9f56-fae03bc5c8ed service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:31:18.614888 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ad74750-db79-482c-9f56-fae03bc5c8ed service neutron] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:31:18.615169 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ad74750-db79-482c-9f56-fae03bc5c8ed service neutron] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:31:18.615441 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ad74750-db79-482c-9f56-fae03bc5c8ed service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:31:18.625785 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ad74750-db79-482c-9f56-fae03bc5c8ed service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:31:18.627240 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ad74750-db79-482c-9f56-fae03bc5c8ed service neutron] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:31:18.637970 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1207/2426] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:31:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:31:18.937585 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21c382b4-6e5c-452f-a700-68b10de4c130 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:31:18.939078 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-21c382b4-6e5c-452f-a700-68b10de4c130 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:31:18.954549 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21c382b4-6e5c-452f-a700-68b10de4c130 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:31:18.960813 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c0fa585-53ff-417e-bfab-4d147585ea1c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:31:18.961575 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c0fa585-53ff-417e-bfab-4d147585ea1c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:31:18.961814 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c0fa585-53ff-417e-bfab-4d147585ea1c admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:31:18.962119 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c0fa585-53ff-417e-bfab-4d147585ea1c admin admin] PATH_INFO: `/v3/projects/7fad852d65d54444b17d925bc42969b5` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:31:18.963438 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c0fa585-53ff-417e-bfab-4d147585ea1c admin admin] RBAC: Authorizing `identity:delete_project(project_id=7fad852d65d54444b17d925bc42969b5)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:31:18.965632 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c0fa585-53ff-417e-bfab-4d147585ea1c admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:31:19.027313 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-3c0fa585-53ff-417e-bfab-4d147585ea1c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7fad852d65d54444b17d925bc42969b5', 'request_id': 'req-3c0fa585-53ff-417e-bfab-4d147585ea1c'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:31:19.034692 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-3c0fa585-53ff-417e-bfab-4d147585ea1c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7fad852d65d54444b17d925bc42969b5', 'request_id': 'req-3c0fa585-53ff-417e-bfab-4d147585ea1c'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:31:19.041999 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-3c0fa585-53ff-417e-bfab-4d147585ea1c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7fad852d65d54444b17d925bc42969b5', 'request_id': 'req-3c0fa585-53ff-417e-bfab-4d147585ea1c'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:31:19.058571 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-3c0fa585-53ff-417e-bfab-4d147585ea1c admin admin] The token cache is being invalidate because project 7fad852d65d54444b17d925bc42969b5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81508) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 01:31:19.058859 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-3c0fa585-53ff-417e-bfab-4d147585ea1c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:31:19.059881 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-3c0fa585-53ff-417e-bfab-4d147585ea1c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:31:19.061480 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1220/2427] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 01:31:18 2022] DELETE /identity/v3/projects/7fad852d65d54444b17d925bc42969b5 => generated 0 bytes in 125 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:31:19.066784 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3325bb49-2552-481a-8a81-a88cd7458794 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:31:19.067087 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3325bb49-2552-481a-8a81-a88cd7458794 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:31:19.067357 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3325bb49-2552-481a-8a81-a88cd7458794 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:31:19.088285 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-3325bb49-2552-481a-8a81-a88cd7458794 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:31:19.089514 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-3325bb49-2552-481a-8a81-a88cd7458794 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:31:19.157060 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:31:19.167541 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1208/2428] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:31:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:31:19.176166 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9533df5-01f9-4930-b02b-b6aaf85a48e8 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:31:19.176535 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9533df5-01f9-4930-b02b-b6aaf85a48e8 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:31:19.176929 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9533df5-01f9-4930-b02b-b6aaf85a48e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:31:19.199062 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-b9533df5-01f9-4930-b02b-b6aaf85a48e8 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:31:19.200298 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-b9533df5-01f9-4930-b02b-b6aaf85a48e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:31:19.209410 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1221/2429] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:31:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:31:19.214081 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3325bb49-2552-481a-8a81-a88cd7458794 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:31:19.225121 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3325bb49-2552-481a-8a81-a88cd7458794 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:31:19.231475 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-331c72c9-4c47-4f3c-bf1a-3d4da00303f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:31:19.232310 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-331c72c9-4c47-4f3c-bf1a-3d4da00303f2 admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:31:19.232629 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-331c72c9-4c47-4f3c-bf1a-3d4da00303f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:31:19.232952 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-331c72c9-4c47-4f3c-bf1a-3d4da00303f2 admin admin] PATH_INFO: `/v3/domains` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:31:19.233971 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-331c72c9-4c47-4f3c-bf1a-3d4da00303f2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 01:31:19.234339 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-331c72c9-4c47-4f3c-bf1a-3d4da00303f2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:31:19.236736 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-331c72c9-4c47-4f3c-bf1a-3d4da00303f2 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:31:19.251387 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1209/2430] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Aug 28 01:31:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 01:31:19.258377 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9533df5-01f9-4930-b02b-b6aaf85a48e8 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:31:19.272491 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9533df5-01f9-4930-b02b-b6aaf85a48e8 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:31:19.278784 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b8782e3-5c12-4642-a651-c2ec530495a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:31:19.279395 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b8782e3-5c12-4642-a651-c2ec530495a0 admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:31:19.279672 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b8782e3-5c12-4642-a651-c2ec530495a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:31:19.279922 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b8782e3-5c12-4642-a651-c2ec530495a0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:31:19.280559 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b8782e3-5c12-4642-a651-c2ec530495a0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:31:19.282534 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b8782e3-5c12-4642-a651-c2ec530495a0 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:31:19.322262 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1222/2431] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 01:31:19 2022] POST /identity/v3/projects => generated 369 bytes in 65 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 01:31:19.327147 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-331c72c9-4c47-4f3c-bf1a-3d4da00303f2 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:31:19.340942 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-331c72c9-4c47-4f3c-bf1a-3d4da00303f2 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:31:19.412534 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e598559b-7392-401d-abfe-6758007ff773 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:31:19.413398 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e598559b-7392-401d-abfe-6758007ff773 admin admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:31:19.413918 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e598559b-7392-401d-abfe-6758007ff773 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:31:19.414204 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e598559b-7392-401d-abfe-6758007ff773 admin admin] PATH_INFO: `/v3/users` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:31:19.414836 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e598559b-7392-401d-abfe-6758007ff773 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:31:19.416766 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e598559b-7392-401d-abfe-6758007ff773 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:31:19.441950 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1210/2432] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 01:31:19 2022] POST /identity/v3/users => generated 285 bytes in 116 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 01:31:19.447245 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b8782e3-5c12-4642-a651-c2ec530495a0 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:31:19.458315 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b8782e3-5c12-4642-a651-c2ec530495a0 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:31:19.463968 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4d79240-b50a-43a0-87d0-5783e6aca65f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:31:19.464605 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d79240-b50a-43a0-87d0-5783e6aca65f admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:31:19.464882 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d79240-b50a-43a0-87d0-5783e6aca65f admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:31:19.465181 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d79240-b50a-43a0-87d0-5783e6aca65f admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:31:19.465832 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4d79240-b50a-43a0-87d0-5783e6aca65f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:31:19.467454 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4d79240-b50a-43a0-87d0-5783e6aca65f admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:31:19.478527 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1223/2433] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 01:31:19 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 01:31:19.482883 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e598559b-7392-401d-abfe-6758007ff773 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:31:19.494034 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e598559b-7392-401d-abfe-6758007ff773 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:31:19.499281 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a84853f-9ad9-4adf-acc7-b4b75de7b0c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:31:19.499887 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a84853f-9ad9-4adf-acc7-b4b75de7b0c2 admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:31:19.500160 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a84853f-9ad9-4adf-acc7-b4b75de7b0c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:31:19.500454 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a84853f-9ad9-4adf-acc7-b4b75de7b0c2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:31:19.501112 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a84853f-9ad9-4adf-acc7-b4b75de7b0c2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:31:19.502723 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a84853f-9ad9-4adf-acc7-b4b75de7b0c2 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:31:19.514517 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1211/2434] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 01:31:19 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 01:31:19.518888 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4d79240-b50a-43a0-87d0-5783e6aca65f admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:31:19.529587 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4d79240-b50a-43a0-87d0-5783e6aca65f admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:31:19.534753 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a500c58d-8489-4948-b6d0-6c18331e5c76 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:31:19.535392 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a500c58d-8489-4948-b6d0-6c18331e5c76 admin admin] REQUEST_METHOD: `PUT` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:31:19.535671 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a500c58d-8489-4948-b6d0-6c18331e5c76 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:31:19.535939 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a500c58d-8489-4948-b6d0-6c18331e5c76 admin admin] PATH_INFO: `/v3/projects/04f5dc0a374b42ceb7130ba0026d60c1/users/2e440a6db0d04b0883e89b143c27505e/roles/0487ef7e943e4701915cb8ce0f2899d7` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:31:19.550498 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a500c58d-8489-4948-b6d0-6c18331e5c76 admin admin] RBAC: Authorizing `identity:create_grant(project_id=04f5dc0a374b42ceb7130ba0026d60c1, user_id=2e440a6db0d04b0883e89b143c27505e, role_id=0487ef7e943e4701915cb8ce0f2899d7)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:31:19.553218 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a500c58d-8489-4948-b6d0-6c18331e5c76 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:31:19.571813 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1224/2435] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 01:31:19 2022] PUT /identity/v3/projects/04f5dc0a374b42ceb7130ba0026d60c1/users/2e440a6db0d04b0883e89b143c27505e/roles/0487ef7e943e4701915cb8ce0f2899d7 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:31:19.583328 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcd66296-3682-4fcb-9bf6-cb25d21d400d None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:31:19.583614 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcd66296-3682-4fcb-9bf6-cb25d21d400d None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:31:19.583887 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcd66296-3682-4fcb-9bf6-cb25d21d400d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:31:19.614690 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-fcd66296-3682-4fcb-9bf6-cb25d21d400d None None] MFA Rules not processed for user `2e440a6db0d04b0883e89b143c27505e`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:31:19.616106 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-fcd66296-3682-4fcb-9bf6-cb25d21d400d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:31:19.684119 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1212/2436] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:31:19 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:31:19.697354 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a500c58d-8489-4948-b6d0-6c18331e5c76 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:31:19.699178 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-a500c58d-8489-4948-b6d0-6c18331e5c76 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:31:19.765614 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:31:19.788431 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a500c58d-8489-4948-b6d0-6c18331e5c76 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:31:19.792322 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09db1198-7eae-45e1-a806-854cd25a689b service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14176b54cf6640abb3b1c14e3824d4f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:31:19.792931 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09db1198-7eae-45e1-a806-854cd25a689b service glance] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:31:19.793239 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09db1198-7eae-45e1-a806-854cd25a689b service glance] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:31:19.793512 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09db1198-7eae-45e1-a806-854cd25a689b service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:31:19.802083 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09db1198-7eae-45e1-a806-854cd25a689b service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:31:19.803581 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09db1198-7eae-45e1-a806-854cd25a689b service glance] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:31:19.812765 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1225/2437] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:31:19 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 116 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:31:23.131657 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcd66296-3682-4fcb-9bf6-cb25d21d400d None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:31:23.133602 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-fcd66296-3682-4fcb-9bf6-cb25d21d400d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:31:23.198290 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcd66296-3682-4fcb-9bf6-cb25d21d400d None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:31:23.202674 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab62e8e1-bc06-43ae-bf43-ae9364d8d368 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:31:23.203305 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab62e8e1-bc06-43ae-bf43-ae9364d8d368 service neutron] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:31:23.203584 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab62e8e1-bc06-43ae-bf43-ae9364d8d368 service neutron] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:31:23.203859 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab62e8e1-bc06-43ae-bf43-ae9364d8d368 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:31:23.205832 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-ab62e8e1-bc06-43ae-bf43-ae9364d8d368 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:31:23.221004 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab62e8e1-bc06-43ae-bf43-ae9364d8d368 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:31:23.223116 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab62e8e1-bc06-43ae-bf43-ae9364d8d368 service neutron] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:31:23.226796 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:31:23.235498 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1213/2438] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:31:23 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:31:29.808594 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=11 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:31:33.236781 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:31:47.102011 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09db1198-7eae-45e1-a806-854cd25a689b service glance] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:31:47.104174 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-09db1198-7eae-45e1-a806-854cd25a689b service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:31:47.166773 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09db1198-7eae-45e1-a806-854cd25a689b service glance] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:31:47.170710 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d65240c-e9de-4136-86cc-6d3c814b531b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:31:47.171324 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d65240c-e9de-4136-86cc-6d3c814b531b service cinder] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:31:47.171600 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d65240c-e9de-4136-86cc-6d3c814b531b service cinder] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:31:47.171869 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d65240c-e9de-4136-86cc-6d3c814b531b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:31:47.173808 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-3d65240c-e9de-4136-86cc-6d3c814b531b service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:31:47.255797 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d65240c-e9de-4136-86cc-6d3c814b531b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:31:47.257206 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d65240c-e9de-4136-86cc-6d3c814b531b service cinder] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:31:47.279454 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1226/2439] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:31:47 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 179 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:31:56.260402 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab62e8e1-bc06-43ae-bf43-ae9364d8d368 service neutron] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:31:56.331766 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab62e8e1-bc06-43ae-bf43-ae9364d8d368 service neutron] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:31:56.336906 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d02a5a9-f3e6-454e-b5a1-31ce63fae4e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:31:56.337571 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d02a5a9-f3e6-454e-b5a1-31ce63fae4e9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:31:56.337848 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d02a5a9-f3e6-454e-b5a1-31ce63fae4e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:31:56.338109 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d02a5a9-f3e6-454e-b5a1-31ce63fae4e9 admin admin] PATH_INFO: `/v3/users/2e440a6db0d04b0883e89b143c27505e` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:31:56.339389 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d02a5a9-f3e6-454e-b5a1-31ce63fae4e9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2e440a6db0d04b0883e89b143c27505e)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:31:56.341362 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d02a5a9-f3e6-454e-b5a1-31ce63fae4e9 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:31:56.399222 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-8d02a5a9-f3e6-454e-b5a1-31ce63fae4e9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2e440a6db0d04b0883e89b143c27505e'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:31:56.413675 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-8d02a5a9-f3e6-454e-b5a1-31ce63fae4e9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2e440a6db0d04b0883e89b143c27505e'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:31:56.424208 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-8d02a5a9-f3e6-454e-b5a1-31ce63fae4e9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2e440a6db0d04b0883e89b143c27505e'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:31:56.432069 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-8d02a5a9-f3e6-454e-b5a1-31ce63fae4e9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2e440a6db0d04b0883e89b143c27505e'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:31:56.432661 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-8d02a5a9-f3e6-454e-b5a1-31ce63fae4e9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2e440a6db0d04b0883e89b143c27505e'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:31:56.438830 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1214/2440] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 01:31:56 2022] DELETE /identity/v3/users/2e440a6db0d04b0883e89b143c27505e => generated 0 bytes in 179 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:31:56.443527 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccb6f9ee-008f-468f-a68d-6e1fc5a77b54 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:31:56.444018 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccb6f9ee-008f-468f-a68d-6e1fc5a77b54 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:31:56.444018 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccb6f9ee-008f-468f-a68d-6e1fc5a77b54 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:31:56.463608 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-ccb6f9ee-008f-468f-a68d-6e1fc5a77b54 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:31:56.464822 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-ccb6f9ee-008f-468f-a68d-6e1fc5a77b54 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:31:56.528881 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1227/2441] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:31:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 28 01:31:56.538770 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d02a5a9-f3e6-454e-b5a1-31ce63fae4e9 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:31:56.540295 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-8d02a5a9-f3e6-454e-b5a1-31ce63fae4e9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:31:56.602313 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d02a5a9-f3e6-454e-b5a1-31ce63fae4e9 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:31:56.606546 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39050619-1a5c-4393-91ee-106544d30841 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:31:56.607157 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39050619-1a5c-4393-91ee-106544d30841 service neutron] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:31:56.607460 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39050619-1a5c-4393-91ee-106544d30841 service neutron] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:31:56.607754 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39050619-1a5c-4393-91ee-106544d30841 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:31:56.617709 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39050619-1a5c-4393-91ee-106544d30841 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:31:56.619312 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39050619-1a5c-4393-91ee-106544d30841 service neutron] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:31:56.628861 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1215/2442] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:31:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:31:56.914854 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccb6f9ee-008f-468f-a68d-6e1fc5a77b54 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:31:56.916374 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-ccb6f9ee-008f-468f-a68d-6e1fc5a77b54 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:31:56.935156 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccb6f9ee-008f-468f-a68d-6e1fc5a77b54 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:31:56.940868 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7352c53a-2bac-41ea-87dc-b33043c26a42 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:31:56.941657 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7352c53a-2bac-41ea-87dc-b33043c26a42 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:31:56.941836 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7352c53a-2bac-41ea-87dc-b33043c26a42 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:31:56.942109 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7352c53a-2bac-41ea-87dc-b33043c26a42 admin admin] PATH_INFO: `/v3/projects/04f5dc0a374b42ceb7130ba0026d60c1` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:31:56.943344 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7352c53a-2bac-41ea-87dc-b33043c26a42 admin admin] RBAC: Authorizing `identity:delete_project(project_id=04f5dc0a374b42ceb7130ba0026d60c1)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:31:56.945341 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7352c53a-2bac-41ea-87dc-b33043c26a42 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:31:56.970171 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:31:56.997046 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-7352c53a-2bac-41ea-87dc-b33043c26a42 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '04f5dc0a374b42ceb7130ba0026d60c1', 'request_id': 'req-7352c53a-2bac-41ea-87dc-b33043c26a42'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:31:57.003824 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-7352c53a-2bac-41ea-87dc-b33043c26a42 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '04f5dc0a374b42ceb7130ba0026d60c1', 'request_id': 'req-7352c53a-2bac-41ea-87dc-b33043c26a42'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:31:57.010502 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-7352c53a-2bac-41ea-87dc-b33043c26a42 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '04f5dc0a374b42ceb7130ba0026d60c1', 'request_id': 'req-7352c53a-2bac-41ea-87dc-b33043c26a42'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:31:57.026170 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-7352c53a-2bac-41ea-87dc-b33043c26a42 admin admin] The token cache is being invalidate because project 04f5dc0a374b42ceb7130ba0026d60c1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81508) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 01:31:57.026460 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-7352c53a-2bac-41ea-87dc-b33043c26a42 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:31:57.027133 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-7352c53a-2bac-41ea-87dc-b33043c26a42 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:31:57.028903 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1228/2443] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 01:31:56 2022] DELETE /identity/v3/projects/04f5dc0a374b42ceb7130ba0026d60c1 => generated 0 bytes in 115 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:31:57.034059 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52caca50-916f-46d9-9654-0cb08d547731 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:31:57.034285 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52caca50-916f-46d9-9654-0cb08d547731 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:31:57.034553 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52caca50-916f-46d9-9654-0cb08d547731 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:31:57.056229 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-52caca50-916f-46d9-9654-0cb08d547731 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:31:57.057466 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-52caca50-916f-46d9-9654-0cb08d547731 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:31:57.076426 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:31:57.121636 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1216/2444] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:31:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:31:57.129843 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e223c9e9-6904-4874-9350-13d3cf37d36a None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:31:57.130098 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e223c9e9-6904-4874-9350-13d3cf37d36a None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:31:57.130431 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e223c9e9-6904-4874-9350-13d3cf37d36a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:31:57.151403 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-e223c9e9-6904-4874-9350-13d3cf37d36a None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:31:57.152596 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-e223c9e9-6904-4874-9350-13d3cf37d36a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:31:57.160880 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1229/2445] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:31:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:31:57.166155 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52caca50-916f-46d9-9654-0cb08d547731 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:31:57.180310 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52caca50-916f-46d9-9654-0cb08d547731 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:31:57.187021 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04f431a8-ef20-4525-91bd-0ea5afca1110 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:31:57.187622 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04f431a8-ef20-4525-91bd-0ea5afca1110 admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:31:57.187893 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04f431a8-ef20-4525-91bd-0ea5afca1110 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:31:57.188155 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04f431a8-ef20-4525-91bd-0ea5afca1110 admin admin] PATH_INFO: `/v3/domains` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:31:57.188782 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04f431a8-ef20-4525-91bd-0ea5afca1110 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 01:31:57.191286 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04f431a8-ef20-4525-91bd-0ea5afca1110 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:31:57.191286 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04f431a8-ef20-4525-91bd-0ea5afca1110 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:31:57.203490 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1217/2446] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Aug 28 01:31:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 01:31:57.208349 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e223c9e9-6904-4874-9350-13d3cf37d36a None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:31:57.218803 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e223c9e9-6904-4874-9350-13d3cf37d36a None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:31:57.224479 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-879d26db-5197-45c5-8344-702b2604ee6a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:31:57.225203 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-879d26db-5197-45c5-8344-702b2604ee6a admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:31:57.225479 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-879d26db-5197-45c5-8344-702b2604ee6a admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:31:57.225754 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-879d26db-5197-45c5-8344-702b2604ee6a admin admin] PATH_INFO: `/v3/projects` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:31:57.226357 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-879d26db-5197-45c5-8344-702b2604ee6a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:31:57.228252 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-879d26db-5197-45c5-8344-702b2604ee6a admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:31:57.258836 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1230/2447] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 01:31:57 2022] POST /identity/v3/projects => generated 365 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 01:31:57.264982 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04f431a8-ef20-4525-91bd-0ea5afca1110 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:31:57.277972 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04f431a8-ef20-4525-91bd-0ea5afca1110 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:31:57.348482 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d2ce74d-3e39-4302-ac3c-13bb04935a9d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:31:57.349133 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d2ce74d-3e39-4302-ac3c-13bb04935a9d admin admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:31:57.349417 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d2ce74d-3e39-4302-ac3c-13bb04935a9d admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:31:57.349695 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d2ce74d-3e39-4302-ac3c-13bb04935a9d admin admin] PATH_INFO: `/v3/users` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:31:57.350393 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d2ce74d-3e39-4302-ac3c-13bb04935a9d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:31:57.352326 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d2ce74d-3e39-4302-ac3c-13bb04935a9d admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:31:57.377275 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1218/2448] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 01:31:57 2022] POST /identity/v3/users => generated 283 bytes in 113 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 01:31:57.381809 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-879d26db-5197-45c5-8344-702b2604ee6a admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:31:57.393843 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-879d26db-5197-45c5-8344-702b2604ee6a admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:31:57.401190 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d83b5593-048d-4b19-be3c-ea121c630d59 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:31:57.401793 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d83b5593-048d-4b19-be3c-ea121c630d59 admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:31:57.402073 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d83b5593-048d-4b19-be3c-ea121c630d59 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:31:57.402333 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d83b5593-048d-4b19-be3c-ea121c630d59 admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:31:57.402969 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d83b5593-048d-4b19-be3c-ea121c630d59 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:31:57.404599 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d83b5593-048d-4b19-be3c-ea121c630d59 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:31:57.415080 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1231/2449] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 01:31:57 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 01:31:57.419240 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d2ce74d-3e39-4302-ac3c-13bb04935a9d admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:31:57.430112 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d2ce74d-3e39-4302-ac3c-13bb04935a9d admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:31:57.436223 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01314a7b-9244-4dc4-a288-15a919809c4f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:31:57.436825 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01314a7b-9244-4dc4-a288-15a919809c4f admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:31:57.437142 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01314a7b-9244-4dc4-a288-15a919809c4f admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:31:57.437417 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01314a7b-9244-4dc4-a288-15a919809c4f admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:31:57.438062 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01314a7b-9244-4dc4-a288-15a919809c4f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:31:57.439668 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01314a7b-9244-4dc4-a288-15a919809c4f admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:31:57.450761 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1219/2450] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 01:31:57 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 01:31:57.458315 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d83b5593-048d-4b19-be3c-ea121c630d59 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:31:57.468994 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d83b5593-048d-4b19-be3c-ea121c630d59 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:31:57.474149 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcbf83c6-35d9-41b1-9237-5a6ea95320a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:31:57.474798 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcbf83c6-35d9-41b1-9237-5a6ea95320a6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:31:57.475162 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcbf83c6-35d9-41b1-9237-5a6ea95320a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:31:57.475440 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcbf83c6-35d9-41b1-9237-5a6ea95320a6 admin admin] PATH_INFO: `/v3/projects/2a0cdb6781014b3db4f8260ecae19d67/users/96c1200f583345938a5e8dce95a9ba4d/roles/0487ef7e943e4701915cb8ce0f2899d7` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:31:57.490406 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcbf83c6-35d9-41b1-9237-5a6ea95320a6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2a0cdb6781014b3db4f8260ecae19d67, user_id=96c1200f583345938a5e8dce95a9ba4d, role_id=0487ef7e943e4701915cb8ce0f2899d7)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:31:57.493381 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcbf83c6-35d9-41b1-9237-5a6ea95320a6 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:31:57.511436 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1232/2451] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 01:31:57 2022] PUT /identity/v3/projects/2a0cdb6781014b3db4f8260ecae19d67/users/96c1200f583345938a5e8dce95a9ba4d/roles/0487ef7e943e4701915cb8ce0f2899d7 => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:31:57.527619 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0680564-8999-4440-9827-ceb2943eb000 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:31:57.528422 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0680564-8999-4440-9827-ceb2943eb000 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:31:57.528422 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0680564-8999-4440-9827-ceb2943eb000 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:31:57.556613 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-d0680564-8999-4440-9827-ceb2943eb000 None None] MFA Rules not processed for user `96c1200f583345938a5e8dce95a9ba4d`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:31:57.557855 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-d0680564-8999-4440-9827-ceb2943eb000 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:31:57.625627 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1220/2452] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:31:57 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 28 01:31:57.637864 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcbf83c6-35d9-41b1-9237-5a6ea95320a6 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:31:57.639870 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-fcbf83c6-35d9-41b1-9237-5a6ea95320a6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:31:57.709009 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcbf83c6-35d9-41b1-9237-5a6ea95320a6 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:31:57.713284 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a731c27-7062-4f81-ac6f-16c8d1ef403e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:31:57.713900 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a731c27-7062-4f81-ac6f-16c8d1ef403e service cinder] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:31:57.714147 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a731c27-7062-4f81-ac6f-16c8d1ef403e service cinder] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:31:57.714432 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a731c27-7062-4f81-ac6f-16c8d1ef403e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:31:57.723993 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a731c27-7062-4f81-ac6f-16c8d1ef403e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:31:57.725557 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a731c27-7062-4f81-ac6f-16c8d1ef403e service cinder] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:31:57.735229 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1233/2453] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:31:57 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:32:07.622155 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=89,keystone:INSERT=3 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:32:07.730912 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:DELETE=7,keystone:SELECT=68,keystone:INSERT=3 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:32:45.931742 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0680564-8999-4440-9827-ceb2943eb000 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:32:45.997099 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0680564-8999-4440-9827-ceb2943eb000 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:32:46.002037 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29166051-d28c-42ce-a5ef-b1a6047da900 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:32:46.002676 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29166051-d28c-42ce-a5ef-b1a6047da900 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:32:46.002953 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29166051-d28c-42ce-a5ef-b1a6047da900 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:32:46.003225 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29166051-d28c-42ce-a5ef-b1a6047da900 admin admin] PATH_INFO: `/v3/users/96c1200f583345938a5e8dce95a9ba4d` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:32:46.004359 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29166051-d28c-42ce-a5ef-b1a6047da900 admin admin] RBAC: Authorizing `identity:delete_user(user_id=96c1200f583345938a5e8dce95a9ba4d)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:32:46.006340 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29166051-d28c-42ce-a5ef-b1a6047da900 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:32:46.056353 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-29166051-d28c-42ce-a5ef-b1a6047da900 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '96c1200f583345938a5e8dce95a9ba4d'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:32:46.070313 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-29166051-d28c-42ce-a5ef-b1a6047da900 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '96c1200f583345938a5e8dce95a9ba4d'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:32:46.079776 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-29166051-d28c-42ce-a5ef-b1a6047da900 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '96c1200f583345938a5e8dce95a9ba4d'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:32:46.087023 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-29166051-d28c-42ce-a5ef-b1a6047da900 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '96c1200f583345938a5e8dce95a9ba4d'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:32:46.087711 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-29166051-d28c-42ce-a5ef-b1a6047da900 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '96c1200f583345938a5e8dce95a9ba4d'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:32:46.093834 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1221/2454] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 01:32:45 2022] DELETE /identity/v3/users/96c1200f583345938a5e8dce95a9ba4d => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:32:46.098448 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63c3cc41-21d9-45a5-b502-21a431dc4bcd None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:32:46.098961 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63c3cc41-21d9-45a5-b502-21a431dc4bcd None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:32:46.098961 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63c3cc41-21d9-45a5-b502-21a431dc4bcd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:32:46.119222 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-63c3cc41-21d9-45a5-b502-21a431dc4bcd None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:32:46.120419 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-63c3cc41-21d9-45a5-b502-21a431dc4bcd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:32:46.187395 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1234/2455] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:32:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:32:46.198201 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29166051-d28c-42ce-a5ef-b1a6047da900 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:32:46.199859 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-29166051-d28c-42ce-a5ef-b1a6047da900 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:32:46.261492 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29166051-d28c-42ce-a5ef-b1a6047da900 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:32:46.265871 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75b6574c-e8dd-48b6-8848-c5e780589293 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:32:46.266489 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75b6574c-e8dd-48b6-8848-c5e780589293 service neutron] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:32:46.266761 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75b6574c-e8dd-48b6-8848-c5e780589293 service neutron] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:32:46.267022 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75b6574c-e8dd-48b6-8848-c5e780589293 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:32:46.277004 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75b6574c-e8dd-48b6-8848-c5e780589293 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:32:46.278456 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75b6574c-e8dd-48b6-8848-c5e780589293 service neutron] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:32:46.288355 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1222/2456] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:32:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:32:46.588472 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63c3cc41-21d9-45a5-b502-21a431dc4bcd None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:32:46.590389 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-63c3cc41-21d9-45a5-b502-21a431dc4bcd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:32:46.603562 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63c3cc41-21d9-45a5-b502-21a431dc4bcd None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:32:46.608931 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baf8ee45-bfdd-4bff-82d2-3f0a12cbd4c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:32:46.609617 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baf8ee45-bfdd-4bff-82d2-3f0a12cbd4c1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:32:46.609885 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baf8ee45-bfdd-4bff-82d2-3f0a12cbd4c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:32:46.610149 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baf8ee45-bfdd-4bff-82d2-3f0a12cbd4c1 admin admin] PATH_INFO: `/v3/projects/2a0cdb6781014b3db4f8260ecae19d67` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:32:46.611427 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baf8ee45-bfdd-4bff-82d2-3f0a12cbd4c1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2a0cdb6781014b3db4f8260ecae19d67)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:32:46.614059 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baf8ee45-bfdd-4bff-82d2-3f0a12cbd4c1 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:32:46.672160 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-baf8ee45-bfdd-4bff-82d2-3f0a12cbd4c1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2a0cdb6781014b3db4f8260ecae19d67', 'request_id': 'req-baf8ee45-bfdd-4bff-82d2-3f0a12cbd4c1'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:32:46.678950 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-baf8ee45-bfdd-4bff-82d2-3f0a12cbd4c1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2a0cdb6781014b3db4f8260ecae19d67', 'request_id': 'req-baf8ee45-bfdd-4bff-82d2-3f0a12cbd4c1'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:32:46.686535 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-baf8ee45-bfdd-4bff-82d2-3f0a12cbd4c1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2a0cdb6781014b3db4f8260ecae19d67', 'request_id': 'req-baf8ee45-bfdd-4bff-82d2-3f0a12cbd4c1'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:32:46.701484 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-baf8ee45-bfdd-4bff-82d2-3f0a12cbd4c1 admin admin] The token cache is being invalidate because project 2a0cdb6781014b3db4f8260ecae19d67 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81508) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 01:32:46.701772 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-baf8ee45-bfdd-4bff-82d2-3f0a12cbd4c1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:32:46.702491 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-baf8ee45-bfdd-4bff-82d2-3f0a12cbd4c1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:32:46.704236 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1235/2457] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 01:32:46 2022] DELETE /identity/v3/projects/2a0cdb6781014b3db4f8260ecae19d67 => generated 0 bytes in 117 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:32:56.284009 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=79,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:32:56.697819 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=54,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:34:29.412732 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75b6574c-e8dd-48b6-8848-c5e780589293 service neutron] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:34:29.415050 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-75b6574c-e8dd-48b6-8848-c5e780589293 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:34:29.519504 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75b6574c-e8dd-48b6-8848-c5e780589293 service neutron] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:34:29.579824 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42790d4e-9b1c-44c2-859b-5004c0817344 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2a3834fa6894dada3291e4781bbd553', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:34:29.580443 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42790d4e-9b1c-44c2-859b-5004c0817344 service placement] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:34:29.580728 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42790d4e-9b1c-44c2-859b-5004c0817344 service placement] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:34:29.580993 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42790d4e-9b1c-44c2-859b-5004c0817344 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:34:29.583128 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-42790d4e-9b1c-44c2-859b-5004c0817344 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:34:29.594484 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42790d4e-9b1c-44c2-859b-5004c0817344 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:34:29.596078 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42790d4e-9b1c-44c2-859b-5004c0817344 service placement] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:34:29.605784 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1223/2458] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:34:29 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 194 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:34:39.601784 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=69 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:35:19.095082 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baf8ee45-bfdd-4bff-82d2-3f0a12cbd4c1 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:35:19.096872 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-baf8ee45-bfdd-4bff-82d2-3f0a12cbd4c1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:35:19.176205 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baf8ee45-bfdd-4bff-82d2-3f0a12cbd4c1 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:35:19.180325 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb3b8bf9-407b-4482-a8b7-240b954cc121 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:35:19.180948 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb3b8bf9-407b-4482-a8b7-240b954cc121 service neutron] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:19.181592 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb3b8bf9-407b-4482-a8b7-240b954cc121 service neutron] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:19.181868 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb3b8bf9-407b-4482-a8b7-240b954cc121 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:19.183822 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-cb3b8bf9-407b-4482-a8b7-240b954cc121 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:35:19.254252 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb3b8bf9-407b-4482-a8b7-240b954cc121 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:35:19.255679 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb3b8bf9-407b-4482-a8b7-240b954cc121 service neutron] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:35:19.267578 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1236/2459] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:35:19 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:35:22.721504 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42790d4e-9b1c-44c2-859b-5004c0817344 service placement] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:35:22.724824 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-42790d4e-9b1c-44c2-859b-5004c0817344 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:35:22.771782 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42790d4e-9b1c-44c2-859b-5004c0817344 service placement] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:35:22.780882 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8d98712-7086-4f88-a088-12c48b7af4ce service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2a3834fa6894dada3291e4781bbd553', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:35:22.782145 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8d98712-7086-4f88-a088-12c48b7af4ce service placement] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:22.782633 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8d98712-7086-4f88-a088-12c48b7af4ce service placement] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:22.783409 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8d98712-7086-4f88-a088-12c48b7af4ce service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:22.784804 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb3b8bf9-407b-4482-a8b7-240b954cc121 service neutron] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:35:22.786726 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-e8d98712-7086-4f88-a088-12c48b7af4ce service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:35:22.786951 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-cb3b8bf9-407b-4482-a8b7-240b954cc121 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:35:22.805912 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8d98712-7086-4f88-a088-12c48b7af4ce service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:35:22.807898 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8d98712-7086-4f88-a088-12c48b7af4ce service placement] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:35:22.811441 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb3b8bf9-407b-4482-a8b7-240b954cc121 service neutron] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:35:22.820634 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e59c988-ae3c-4841-b922-5013a0c9bcb2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:35:22.821423 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e59c988-ae3c-4841-b922-5013a0c9bcb2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:22.821698 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e59c988-ae3c-4841-b922-5013a0c9bcb2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:22.821994 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e59c988-ae3c-4841-b922-5013a0c9bcb2 admin admin] PATH_INFO: `/v3/users/d209174d0db04b3584556824c6d0a3d4` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:22.823126 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e59c988-ae3c-4841-b922-5013a0c9bcb2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d209174d0db04b3584556824c6d0a3d4)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:35:22.825138 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1224/2460] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:35:22 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:35:22.825297 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e59c988-ae3c-4841-b922-5013a0c9bcb2 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:35:22.876804 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-1e59c988-ae3c-4841-b922-5013a0c9bcb2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd209174d0db04b3584556824c6d0a3d4'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:35:22.877484 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-1e59c988-ae3c-4841-b922-5013a0c9bcb2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd209174d0db04b3584556824c6d0a3d4'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:35:22.887265 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-1e59c988-ae3c-4841-b922-5013a0c9bcb2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd209174d0db04b3584556824c6d0a3d4'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:35:22.895169 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-1e59c988-ae3c-4841-b922-5013a0c9bcb2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd209174d0db04b3584556824c6d0a3d4'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:35:22.895788 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-1e59c988-ae3c-4841-b922-5013a0c9bcb2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd209174d0db04b3584556824c6d0a3d4'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:35:22.913685 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1237/2461] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 01:35:22 2022] DELETE /identity/v3/users/d209174d0db04b3584556824c6d0a3d4 => generated 0 bytes in 129 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 28 01:35:23.101204 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8d98712-7086-4f88-a088-12c48b7af4ce service placement] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:35:23.117146 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8d98712-7086-4f88-a088-12c48b7af4ce service placement] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:35:23.176567 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d41fc3c3-65b2-4730-932a-ae8e9a72cc53 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:35:23.177141 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d41fc3c3-65b2-4730-932a-ae8e9a72cc53 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:23.177399 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d41fc3c3-65b2-4730-932a-ae8e9a72cc53 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:23.177680 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d41fc3c3-65b2-4730-932a-ae8e9a72cc53 admin admin] PATH_INFO: `/v3/projects/95df81e14a8549fca0851234f4a00b89` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:23.178787 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d41fc3c3-65b2-4730-932a-ae8e9a72cc53 admin admin] RBAC: Authorizing `identity:delete_project(project_id=95df81e14a8549fca0851234f4a00b89)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:35:23.180695 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d41fc3c3-65b2-4730-932a-ae8e9a72cc53 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:35:23.233010 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-d41fc3c3-65b2-4730-932a-ae8e9a72cc53 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '95df81e14a8549fca0851234f4a00b89', 'request_id': 'req-d41fc3c3-65b2-4730-932a-ae8e9a72cc53'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:35:23.238537 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-d41fc3c3-65b2-4730-932a-ae8e9a72cc53 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '95df81e14a8549fca0851234f4a00b89', 'request_id': 'req-d41fc3c3-65b2-4730-932a-ae8e9a72cc53'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:35:23.245313 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-d41fc3c3-65b2-4730-932a-ae8e9a72cc53 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '95df81e14a8549fca0851234f4a00b89', 'request_id': 'req-d41fc3c3-65b2-4730-932a-ae8e9a72cc53'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:35:23.265212 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-d41fc3c3-65b2-4730-932a-ae8e9a72cc53 admin admin] The token cache is being invalidate because project 95df81e14a8549fca0851234f4a00b89 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81507) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 01:35:23.265520 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-d41fc3c3-65b2-4730-932a-ae8e9a72cc53 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:35:23.266548 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-d41fc3c3-65b2-4730-932a-ae8e9a72cc53 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:35:23.268194 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1225/2462] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 01:35:23 2022] DELETE /identity/v3/projects/95df81e14a8549fca0851234f4a00b89 => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:35:23.274244 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0e76081-c45d-4737-996f-532423ca937f None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:23.274543 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0e76081-c45d-4737-996f-532423ca937f None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:23.274801 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0e76081-c45d-4737-996f-532423ca937f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:23.295399 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-b0e76081-c45d-4737-996f-532423ca937f None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:35:23.296630 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-b0e76081-c45d-4737-996f-532423ca937f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:35:23.306438 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:35:23.359592 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1238/2463] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:35:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:35:23.368799 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd5ac68f-cd9d-41c4-a534-e0a03aa4d382 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:23.369203 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd5ac68f-cd9d-41c4-a534-e0a03aa4d382 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:23.369570 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd5ac68f-cd9d-41c4-a534-e0a03aa4d382 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:23.395305 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-fd5ac68f-cd9d-41c4-a534-e0a03aa4d382 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:35:23.396514 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-fd5ac68f-cd9d-41c4-a534-e0a03aa4d382 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:35:23.404930 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1226/2464] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:35:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:35:23.409522 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0e76081-c45d-4737-996f-532423ca937f None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:35:23.419551 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0e76081-c45d-4737-996f-532423ca937f None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:35:23.425312 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c40d4423-f871-4bac-8c33-7040c296a075 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:35:23.425885 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c40d4423-f871-4bac-8c33-7040c296a075 admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:23.426203 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c40d4423-f871-4bac-8c33-7040c296a075 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:23.426467 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c40d4423-f871-4bac-8c33-7040c296a075 admin admin] PATH_INFO: `/v3/domains` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:23.427177 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c40d4423-f871-4bac-8c33-7040c296a075 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 01:35:23.427413 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c40d4423-f871-4bac-8c33-7040c296a075 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:35:23.429044 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c40d4423-f871-4bac-8c33-7040c296a075 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:35:23.441031 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1239/2465] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Aug 28 01:35:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 01:35:23.447039 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd5ac68f-cd9d-41c4-a534-e0a03aa4d382 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:35:23.457012 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd5ac68f-cd9d-41c4-a534-e0a03aa4d382 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:35:23.461931 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c5de903-a36e-431f-8351-afbadada121e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:35:23.462525 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c5de903-a36e-431f-8351-afbadada121e admin admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:23.462796 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c5de903-a36e-431f-8351-afbadada121e admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:23.463057 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c5de903-a36e-431f-8351-afbadada121e admin admin] PATH_INFO: `/v3/projects` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:23.463683 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c5de903-a36e-431f-8351-afbadada121e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:35:23.465618 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c5de903-a36e-431f-8351-afbadada121e admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:35:23.495251 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1227/2466] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 01:35:23 2022] POST /identity/v3/projects => generated 387 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 01:35:23.499702 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c40d4423-f871-4bac-8c33-7040c296a075 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:35:23.510277 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c40d4423-f871-4bac-8c33-7040c296a075 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:35:23.582855 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63cde11a-62e0-4b99-8577-a90229e12617 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:35:23.583686 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63cde11a-62e0-4b99-8577-a90229e12617 admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:23.584060 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63cde11a-62e0-4b99-8577-a90229e12617 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:23.584390 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63cde11a-62e0-4b99-8577-a90229e12617 admin admin] PATH_INFO: `/v3/users` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:23.585377 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63cde11a-62e0-4b99-8577-a90229e12617 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:35:23.587871 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63cde11a-62e0-4b99-8577-a90229e12617 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:35:23.614347 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1240/2467] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 01:35:23 2022] POST /identity/v3/users => generated 294 bytes in 115 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 01:35:23.618693 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c5de903-a36e-431f-8351-afbadada121e admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:35:23.629448 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c5de903-a36e-431f-8351-afbadada121e admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:35:23.635109 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f992ede6-01a1-4bc7-a7a8-48c64f4a6d6e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:35:23.635708 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f992ede6-01a1-4bc7-a7a8-48c64f4a6d6e admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:23.635992 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f992ede6-01a1-4bc7-a7a8-48c64f4a6d6e admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:23.636254 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f992ede6-01a1-4bc7-a7a8-48c64f4a6d6e admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:23.636886 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f992ede6-01a1-4bc7-a7a8-48c64f4a6d6e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:35:23.638530 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f992ede6-01a1-4bc7-a7a8-48c64f4a6d6e admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:35:23.648821 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1228/2468] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 01:35:23 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 01:35:23.652873 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63cde11a-62e0-4b99-8577-a90229e12617 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:35:23.663384 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63cde11a-62e0-4b99-8577-a90229e12617 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:35:23.668947 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77d617db-2d45-488d-ab7a-6c3655480cf8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:35:23.669572 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77d617db-2d45-488d-ab7a-6c3655480cf8 admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:23.669849 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77d617db-2d45-488d-ab7a-6c3655480cf8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:23.670134 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77d617db-2d45-488d-ab7a-6c3655480cf8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:23.670856 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77d617db-2d45-488d-ab7a-6c3655480cf8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:35:23.672496 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77d617db-2d45-488d-ab7a-6c3655480cf8 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:35:23.684920 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1241/2469] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 01:35:23 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 01:35:23.689241 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f992ede6-01a1-4bc7-a7a8-48c64f4a6d6e admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:35:23.699428 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f992ede6-01a1-4bc7-a7a8-48c64f4a6d6e admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:35:23.704561 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57a041a6-5608-4505-810c-e5ee7892f0b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:35:23.705253 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57a041a6-5608-4505-810c-e5ee7892f0b4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:23.705526 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57a041a6-5608-4505-810c-e5ee7892f0b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:23.705795 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57a041a6-5608-4505-810c-e5ee7892f0b4 admin admin] PATH_INFO: `/v3/projects/d59af4a3e51543ecaeae5c83dd43df7e/users/0be295124d3946bfb7089d31c537a8a2/roles/0487ef7e943e4701915cb8ce0f2899d7` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:23.721027 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57a041a6-5608-4505-810c-e5ee7892f0b4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d59af4a3e51543ecaeae5c83dd43df7e, user_id=0be295124d3946bfb7089d31c537a8a2, role_id=0487ef7e943e4701915cb8ce0f2899d7)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:35:23.723678 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57a041a6-5608-4505-810c-e5ee7892f0b4 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:35:23.740701 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1229/2470] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 01:35:23 2022] PUT /identity/v3/projects/d59af4a3e51543ecaeae5c83dd43df7e/users/0be295124d3946bfb7089d31c537a8a2/roles/0487ef7e943e4701915cb8ce0f2899d7 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:35:23.751123 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73068925-fcd2-478c-a6bc-011a5a4b67ba None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:23.751564 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73068925-fcd2-478c-a6bc-011a5a4b67ba None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:23.751836 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73068925-fcd2-478c-a6bc-011a5a4b67ba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:23.780924 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-73068925-fcd2-478c-a6bc-011a5a4b67ba None None] MFA Rules not processed for user `0be295124d3946bfb7089d31c537a8a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:35:23.782178 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-73068925-fcd2-478c-a6bc-011a5a4b67ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:35:23.851619 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1242/2471] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:35:23 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:35:23.856279 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57a041a6-5608-4505-810c-e5ee7892f0b4 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:35:23.862323 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:35:23.923629 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57a041a6-5608-4505-810c-e5ee7892f0b4 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:35:23.928794 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d6af490-f3ca-457c-aab2-7e9068c23beb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:35:23.929429 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d6af490-f3ca-457c-aab2-7e9068c23beb admin admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:23.929710 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d6af490-f3ca-457c-aab2-7e9068c23beb admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:23.929978 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d6af490-f3ca-457c-aab2-7e9068c23beb admin admin] PATH_INFO: `/v3/projects` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:23.930616 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d6af490-f3ca-457c-aab2-7e9068c23beb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:35:23.932860 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d6af490-f3ca-457c-aab2-7e9068c23beb admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:35:23.962121 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1230/2472] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 01:35:23 2022] POST /identity/v3/projects => generated 385 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 01:35:23.966391 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73068925-fcd2-478c-a6bc-011a5a4b67ba None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:35:23.978456 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73068925-fcd2-478c-a6bc-011a5a4b67ba None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:35:23.983616 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:35:24.042051 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8d07064-743b-41a6-9765-a8ffd6137fda admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:35:24.042678 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8d07064-743b-41a6-9765-a8ffd6137fda admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:24.042958 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8d07064-743b-41a6-9765-a8ffd6137fda admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:24.043226 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8d07064-743b-41a6-9765-a8ffd6137fda admin admin] PATH_INFO: `/v3/users` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:24.043862 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8d07064-743b-41a6-9765-a8ffd6137fda admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:35:24.045927 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8d07064-743b-41a6-9765-a8ffd6137fda admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:35:24.069538 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1243/2473] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 01:35:23 2022] POST /identity/v3/users => generated 299 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 01:35:24.073785 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d6af490-f3ca-457c-aab2-7e9068c23beb admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:35:24.084060 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d6af490-f3ca-457c-aab2-7e9068c23beb admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:35:24.088835 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63f14e07-60e7-4242-8478-d2781e7044be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:35:24.089464 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63f14e07-60e7-4242-8478-d2781e7044be admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:24.089741 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63f14e07-60e7-4242-8478-d2781e7044be admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:24.090004 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63f14e07-60e7-4242-8478-d2781e7044be admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:24.090609 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63f14e07-60e7-4242-8478-d2781e7044be admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:35:24.092250 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63f14e07-60e7-4242-8478-d2781e7044be admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:35:24.103139 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1231/2474] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 01:35:24 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 01:35:24.107422 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8d07064-743b-41a6-9765-a8ffd6137fda admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:35:24.118552 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8d07064-743b-41a6-9765-a8ffd6137fda admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:35:24.124122 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74d958e1-d978-45e7-988e-4a172827d31e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:35:24.124790 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74d958e1-d978-45e7-988e-4a172827d31e admin admin] REQUEST_METHOD: `PUT` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:24.125111 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74d958e1-d978-45e7-988e-4a172827d31e admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:24.125385 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74d958e1-d978-45e7-988e-4a172827d31e admin admin] PATH_INFO: `/v3/projects/9476c9fc65ad4cc7943c3a5323df5d33/users/215cd5e63b144162849d8f59071eb72f/roles/f08d6043261747e3b4763c744eef7dfb` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:24.142098 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74d958e1-d978-45e7-988e-4a172827d31e admin admin] RBAC: Authorizing `identity:create_grant(project_id=9476c9fc65ad4cc7943c3a5323df5d33, user_id=215cd5e63b144162849d8f59071eb72f, role_id=f08d6043261747e3b4763c744eef7dfb)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:35:24.144820 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74d958e1-d978-45e7-988e-4a172827d31e admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:35:24.161596 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1244/2475] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 01:35:24 2022] PUT /identity/v3/projects/9476c9fc65ad4cc7943c3a5323df5d33/users/215cd5e63b144162849d8f59071eb72f/roles/f08d6043261747e3b4763c744eef7dfb => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:35:24.165802 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63f14e07-60e7-4242-8478-d2781e7044be admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:35:24.176570 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63f14e07-60e7-4242-8478-d2781e7044be admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:35:24.232604 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cc18247-bf2b-42af-bb42-1001e4d1c8c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:35:24.233158 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc18247-bf2b-42af-bb42-1001e4d1c8c0 admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:24.233462 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc18247-bf2b-42af-bb42-1001e4d1c8c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:24.233728 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc18247-bf2b-42af-bb42-1001e4d1c8c0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:24.234354 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cc18247-bf2b-42af-bb42-1001e4d1c8c0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:35:24.236066 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cc18247-bf2b-42af-bb42-1001e4d1c8c0 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:35:24.245833 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1232/2476] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 01:35:24 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 01:35:24.249986 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74d958e1-d978-45e7-988e-4a172827d31e admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:35:24.260414 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74d958e1-d978-45e7-988e-4a172827d31e admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:35:24.265612 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b25adec6-3fc1-40a8-b649-3586f2427afc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:35:24.266357 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b25adec6-3fc1-40a8-b649-3586f2427afc admin admin] REQUEST_METHOD: `PUT` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:24.266658 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b25adec6-3fc1-40a8-b649-3586f2427afc admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:24.266926 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b25adec6-3fc1-40a8-b649-3586f2427afc admin admin] PATH_INFO: `/v3/projects/9476c9fc65ad4cc7943c3a5323df5d33/users/215cd5e63b144162849d8f59071eb72f/roles/f08d6043261747e3b4763c744eef7dfb` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:24.268867 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b25adec6-3fc1-40a8-b649-3586f2427afc admin admin] RBAC: Authorizing `identity:create_grant(project_id=9476c9fc65ad4cc7943c3a5323df5d33, user_id=215cd5e63b144162849d8f59071eb72f, role_id=f08d6043261747e3b4763c744eef7dfb)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:35:24.271568 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b25adec6-3fc1-40a8-b649-3586f2427afc admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:35:24.286087 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1245/2477] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 01:35:24 2022] PUT /identity/v3/projects/9476c9fc65ad4cc7943c3a5323df5d33/users/215cd5e63b144162849d8f59071eb72f/roles/f08d6043261747e3b4763c744eef7dfb => generated 0 bytes in 37 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:35:24.297960 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a4aa2a5-6401-4d95-bffa-ee42e86e2a1c None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:24.298360 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a4aa2a5-6401-4d95-bffa-ee42e86e2a1c None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:24.298783 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a4aa2a5-6401-4d95-bffa-ee42e86e2a1c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:24.331686 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-1a4aa2a5-6401-4d95-bffa-ee42e86e2a1c None None] MFA Rules not processed for user `215cd5e63b144162849d8f59071eb72f`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:35:24.332913 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-1a4aa2a5-6401-4d95-bffa-ee42e86e2a1c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:35:24.408986 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:35:24.413457 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1233/2478] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:35:24 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:35:24.425158 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b25adec6-3fc1-40a8-b649-3586f2427afc admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:35:24.427000 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-b25adec6-3fc1-40a8-b649-3586f2427afc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:35:24.490239 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b25adec6-3fc1-40a8-b649-3586f2427afc admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:35:24.494237 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f04c0e1b-87c3-48fa-a1a7-469df2631cc9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:35:24.494842 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f04c0e1b-87c3-48fa-a1a7-469df2631cc9 service cinder] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:24.495119 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f04c0e1b-87c3-48fa-a1a7-469df2631cc9 service cinder] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:24.495386 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f04c0e1b-87c3-48fa-a1a7-469df2631cc9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:24.505068 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f04c0e1b-87c3-48fa-a1a7-469df2631cc9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:35:24.506586 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f04c0e1b-87c3-48fa-a1a7-469df2631cc9 service cinder] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:35:24.516170 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1246/2479] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:35:24 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:35:24.588834 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a4aa2a5-6401-4d95-bffa-ee42e86e2a1c None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:35:24.655646 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a4aa2a5-6401-4d95-bffa-ee42e86e2a1c None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:35:24.661010 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecf29376-41e9-4acc-a43d-2dd6bcd931f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:35:24.661667 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf29376-41e9-4acc-a43d-2dd6bcd931f4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:24.661941 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf29376-41e9-4acc-a43d-2dd6bcd931f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:24.662212 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf29376-41e9-4acc-a43d-2dd6bcd931f4 admin admin] PATH_INFO: `/v3/users/0be295124d3946bfb7089d31c537a8a2` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:24.663389 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecf29376-41e9-4acc-a43d-2dd6bcd931f4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0be295124d3946bfb7089d31c537a8a2)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:35:24.665361 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecf29376-41e9-4acc-a43d-2dd6bcd931f4 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:35:24.716786 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-ecf29376-41e9-4acc-a43d-2dd6bcd931f4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0be295124d3946bfb7089d31c537a8a2'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:35:24.732650 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-ecf29376-41e9-4acc-a43d-2dd6bcd931f4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0be295124d3946bfb7089d31c537a8a2'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:35:24.743205 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-ecf29376-41e9-4acc-a43d-2dd6bcd931f4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0be295124d3946bfb7089d31c537a8a2'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:35:24.752869 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-ecf29376-41e9-4acc-a43d-2dd6bcd931f4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0be295124d3946bfb7089d31c537a8a2'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:35:24.753621 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-ecf29376-41e9-4acc-a43d-2dd6bcd931f4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0be295124d3946bfb7089d31c537a8a2'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:35:24.762468 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1234/2480] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 01:35:24 2022] DELETE /identity/v3/users/0be295124d3946bfb7089d31c537a8a2 => generated 0 bytes in 174 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:35:24.767256 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceeadde4-4bcf-4976-bbc5-f065d96c08ce None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:24.767547 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceeadde4-4bcf-4976-bbc5-f065d96c08ce None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:24.767814 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceeadde4-4bcf-4976-bbc5-f065d96c08ce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:24.787630 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-ceeadde4-4bcf-4976-bbc5-f065d96c08ce None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:35:24.788839 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-ceeadde4-4bcf-4976-bbc5-f065d96c08ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:35:24.851728 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1247/2481] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:35:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:35:24.863377 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecf29376-41e9-4acc-a43d-2dd6bcd931f4 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:35:24.864987 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-ecf29376-41e9-4acc-a43d-2dd6bcd931f4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:35:24.925417 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecf29376-41e9-4acc-a43d-2dd6bcd931f4 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:35:24.929548 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a47e2bd-18f1-455b-b152-c92ca1a1039c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:35:24.930170 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a47e2bd-18f1-455b-b152-c92ca1a1039c service neutron] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:24.930446 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a47e2bd-18f1-455b-b152-c92ca1a1039c service neutron] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:24.930714 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a47e2bd-18f1-455b-b152-c92ca1a1039c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:24.941513 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a47e2bd-18f1-455b-b152-c92ca1a1039c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:35:24.942948 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a47e2bd-18f1-455b-b152-c92ca1a1039c service neutron] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:35:24.952869 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1235/2482] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:35:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:35:25.235187 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceeadde4-4bcf-4976-bbc5-f065d96c08ce None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:35:25.236722 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-ceeadde4-4bcf-4976-bbc5-f065d96c08ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:35:25.246539 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:35:25.250373 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceeadde4-4bcf-4976-bbc5-f065d96c08ce None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:35:25.256712 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c90f1631-53bf-4621-a757-7197c475f10c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:35:25.257408 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c90f1631-53bf-4621-a757-7197c475f10c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:25.257679 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c90f1631-53bf-4621-a757-7197c475f10c admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:25.257944 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c90f1631-53bf-4621-a757-7197c475f10c admin admin] PATH_INFO: `/v3/projects/d59af4a3e51543ecaeae5c83dd43df7e` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:25.259065 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c90f1631-53bf-4621-a757-7197c475f10c admin admin] RBAC: Authorizing `identity:delete_project(project_id=d59af4a3e51543ecaeae5c83dd43df7e)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:35:25.261018 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c90f1631-53bf-4621-a757-7197c475f10c admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:35:25.311806 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-c90f1631-53bf-4621-a757-7197c475f10c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd59af4a3e51543ecaeae5c83dd43df7e', 'request_id': 'req-c90f1631-53bf-4621-a757-7197c475f10c'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:35:25.317998 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-c90f1631-53bf-4621-a757-7197c475f10c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd59af4a3e51543ecaeae5c83dd43df7e', 'request_id': 'req-c90f1631-53bf-4621-a757-7197c475f10c'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:35:25.324856 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-c90f1631-53bf-4621-a757-7197c475f10c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd59af4a3e51543ecaeae5c83dd43df7e', 'request_id': 'req-c90f1631-53bf-4621-a757-7197c475f10c'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:35:25.340289 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-c90f1631-53bf-4621-a757-7197c475f10c admin admin] The token cache is being invalidate because project d59af4a3e51543ecaeae5c83dd43df7e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81508) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 01:35:25.340671 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-c90f1631-53bf-4621-a757-7197c475f10c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:35:25.341459 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-c90f1631-53bf-4621-a757-7197c475f10c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:35:25.343125 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1248/2483] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 01:35:25 2022] DELETE /identity/v3/projects/d59af4a3e51543ecaeae5c83dd43df7e => generated 0 bytes in 109 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:35:25.352584 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a47e2bd-18f1-455b-b152-c92ca1a1039c service neutron] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:35:25.362585 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a47e2bd-18f1-455b-b152-c92ca1a1039c service neutron] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:35:25.380768 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:35:25.421082 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4951920-d39c-49ea-9b38-de98b64e3451 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:35:25.421742 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4951920-d39c-49ea-9b38-de98b64e3451 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:25.422014 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4951920-d39c-49ea-9b38-de98b64e3451 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:25.422279 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4951920-d39c-49ea-9b38-de98b64e3451 admin admin] PATH_INFO: `/v3/users/215cd5e63b144162849d8f59071eb72f` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:25.423441 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4951920-d39c-49ea-9b38-de98b64e3451 admin admin] RBAC: Authorizing `identity:delete_user(user_id=215cd5e63b144162849d8f59071eb72f)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:35:25.425406 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4951920-d39c-49ea-9b38-de98b64e3451 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:35:25.479727 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-c4951920-d39c-49ea-9b38-de98b64e3451 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '215cd5e63b144162849d8f59071eb72f'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:35:25.491713 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-c4951920-d39c-49ea-9b38-de98b64e3451 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '215cd5e63b144162849d8f59071eb72f'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:35:25.499632 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-c4951920-d39c-49ea-9b38-de98b64e3451 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '215cd5e63b144162849d8f59071eb72f'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:35:25.506617 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-c4951920-d39c-49ea-9b38-de98b64e3451 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '215cd5e63b144162849d8f59071eb72f'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:35:25.507259 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-c4951920-d39c-49ea-9b38-de98b64e3451 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '215cd5e63b144162849d8f59071eb72f'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:35:25.513129 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1236/2484] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 01:35:25 2022] DELETE /identity/v3/users/215cd5e63b144162849d8f59071eb72f => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:35:25.819060 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c90f1631-53bf-4621-a757-7197c475f10c admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:35:25.820753 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-c90f1631-53bf-4621-a757-7197c475f10c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:35:25.836100 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c90f1631-53bf-4621-a757-7197c475f10c admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:35:25.899928 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-372ee456-d968-49de-9e37-49f6c029cce2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:35:25.900628 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-372ee456-d968-49de-9e37-49f6c029cce2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:25.900847 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-372ee456-d968-49de-9e37-49f6c029cce2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:25.901153 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-372ee456-d968-49de-9e37-49f6c029cce2 admin admin] PATH_INFO: `/v3/projects/9476c9fc65ad4cc7943c3a5323df5d33` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:25.902388 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-372ee456-d968-49de-9e37-49f6c029cce2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9476c9fc65ad4cc7943c3a5323df5d33)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:35:25.904338 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-372ee456-d968-49de-9e37-49f6c029cce2 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:35:25.957083 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-372ee456-d968-49de-9e37-49f6c029cce2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9476c9fc65ad4cc7943c3a5323df5d33', 'request_id': 'req-372ee456-d968-49de-9e37-49f6c029cce2'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:35:25.963174 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-372ee456-d968-49de-9e37-49f6c029cce2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9476c9fc65ad4cc7943c3a5323df5d33', 'request_id': 'req-372ee456-d968-49de-9e37-49f6c029cce2'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:35:25.969519 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-372ee456-d968-49de-9e37-49f6c029cce2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9476c9fc65ad4cc7943c3a5323df5d33', 'request_id': 'req-372ee456-d968-49de-9e37-49f6c029cce2'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:35:25.985222 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-372ee456-d968-49de-9e37-49f6c029cce2 admin admin] The token cache is being invalidate because project 9476c9fc65ad4cc7943c3a5323df5d33 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81508) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 01:35:25.985517 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-372ee456-d968-49de-9e37-49f6c029cce2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:35:25.986152 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-372ee456-d968-49de-9e37-49f6c029cce2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:35:25.987664 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1249/2485] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 01:35:25 2022] DELETE /identity/v3/projects/9476c9fc65ad4cc7943c3a5323df5d33 => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:35:25.992774 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1326920f-b8eb-45d1-bbdb-610c27a20002 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:25.993322 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1326920f-b8eb-45d1-bbdb-610c27a20002 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:25.993322 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1326920f-b8eb-45d1-bbdb-610c27a20002 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:26.013372 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-1326920f-b8eb-45d1-bbdb-610c27a20002 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:35:26.014589 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-1326920f-b8eb-45d1-bbdb-610c27a20002 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:35:26.082113 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1237/2486] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:35:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 28 01:35:26.091414 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9403809e-e96a-492d-8bb4-327b48bb9d61 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:26.091691 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9403809e-e96a-492d-8bb4-327b48bb9d61 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:26.091969 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9403809e-e96a-492d-8bb4-327b48bb9d61 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:26.113875 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-9403809e-e96a-492d-8bb4-327b48bb9d61 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:35:26.114916 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-9403809e-e96a-492d-8bb4-327b48bb9d61 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:35:26.124452 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1250/2487] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:35:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:35:26.129000 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1326920f-b8eb-45d1-bbdb-610c27a20002 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:35:26.139492 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1326920f-b8eb-45d1-bbdb-610c27a20002 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:35:26.145356 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6d8f0d9-a437-4228-bdff-7823458f22c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:35:26.146049 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6d8f0d9-a437-4228-bdff-7823458f22c5 admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:26.146393 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6d8f0d9-a437-4228-bdff-7823458f22c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:26.146767 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6d8f0d9-a437-4228-bdff-7823458f22c5 admin admin] PATH_INFO: `/v3/domains` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:26.147493 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6d8f0d9-a437-4228-bdff-7823458f22c5 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 01:35:26.147898 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6d8f0d9-a437-4228-bdff-7823458f22c5 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:35:26.149700 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6d8f0d9-a437-4228-bdff-7823458f22c5 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:35:26.162537 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1238/2488] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Aug 28 01:35:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 01:35:26.167555 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9403809e-e96a-492d-8bb4-327b48bb9d61 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:35:26.179023 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9403809e-e96a-492d-8bb4-327b48bb9d61 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:35:26.185166 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d318b3e6-b199-4dc2-903f-8ebb2ec55ac9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:35:26.185777 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d318b3e6-b199-4dc2-903f-8ebb2ec55ac9 admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:26.186045 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d318b3e6-b199-4dc2-903f-8ebb2ec55ac9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:26.186307 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d318b3e6-b199-4dc2-903f-8ebb2ec55ac9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:26.186927 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d318b3e6-b199-4dc2-903f-8ebb2ec55ac9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:35:26.188827 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d318b3e6-b199-4dc2-903f-8ebb2ec55ac9 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:35:26.198752 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:35:26.218985 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1251/2489] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 01:35:26 2022] POST /identity/v3/projects => generated 387 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 01:35:26.223377 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6d8f0d9-a437-4228-bdff-7823458f22c5 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:35:26.233729 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6d8f0d9-a437-4228-bdff-7823458f22c5 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:35:26.261076 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:35:26.295995 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfa6220e-bc55-40d5-94b1-f807f68fb85f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:35:26.296599 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfa6220e-bc55-40d5-94b1-f807f68fb85f admin admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:26.296875 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfa6220e-bc55-40d5-94b1-f807f68fb85f admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:26.297167 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfa6220e-bc55-40d5-94b1-f807f68fb85f admin admin] PATH_INFO: `/v3/users` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:26.297788 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfa6220e-bc55-40d5-94b1-f807f68fb85f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:35:26.299764 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfa6220e-bc55-40d5-94b1-f807f68fb85f admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:35:26.320979 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1239/2490] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 01:35:26 2022] POST /identity/v3/users => generated 294 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 01:35:26.325078 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d318b3e6-b199-4dc2-903f-8ebb2ec55ac9 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:35:26.336394 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d318b3e6-b199-4dc2-903f-8ebb2ec55ac9 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:35:26.343206 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5088c7e-f0bc-4ffe-b629-aa0d45c6f709 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:35:26.344110 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5088c7e-f0bc-4ffe-b629-aa0d45c6f709 admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:26.344498 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5088c7e-f0bc-4ffe-b629-aa0d45c6f709 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:26.344881 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5088c7e-f0bc-4ffe-b629-aa0d45c6f709 admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:26.345919 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5088c7e-f0bc-4ffe-b629-aa0d45c6f709 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:35:26.348379 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5088c7e-f0bc-4ffe-b629-aa0d45c6f709 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:35:26.365422 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1252/2491] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 01:35:26 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 01:35:26.370498 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfa6220e-bc55-40d5-94b1-f807f68fb85f admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:35:26.382459 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfa6220e-bc55-40d5-94b1-f807f68fb85f admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:35:26.387968 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17bc1dd3-cece-46d2-9e99-bb7813c37d9c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:35:26.388614 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17bc1dd3-cece-46d2-9e99-bb7813c37d9c admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:26.388924 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17bc1dd3-cece-46d2-9e99-bb7813c37d9c admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:26.389208 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17bc1dd3-cece-46d2-9e99-bb7813c37d9c admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:26.389863 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17bc1dd3-cece-46d2-9e99-bb7813c37d9c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:35:26.391620 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17bc1dd3-cece-46d2-9e99-bb7813c37d9c admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:35:26.403109 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1240/2492] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 01:35:26 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 01:35:26.407902 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5088c7e-f0bc-4ffe-b629-aa0d45c6f709 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:35:26.423729 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5088c7e-f0bc-4ffe-b629-aa0d45c6f709 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:35:26.431596 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ffb4501-b0bf-4fd8-8284-97a7ad2d1e49 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:35:26.432261 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ffb4501-b0bf-4fd8-8284-97a7ad2d1e49 admin admin] REQUEST_METHOD: `PUT` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:26.432537 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ffb4501-b0bf-4fd8-8284-97a7ad2d1e49 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:26.432805 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ffb4501-b0bf-4fd8-8284-97a7ad2d1e49 admin admin] PATH_INFO: `/v3/projects/d22b44e41e7c46ba9e2e32fff048ade1/users/ddeeffe4db39448f8f8bc6279c70bfae/roles/0487ef7e943e4701915cb8ce0f2899d7` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:26.448921 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ffb4501-b0bf-4fd8-8284-97a7ad2d1e49 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d22b44e41e7c46ba9e2e32fff048ade1, user_id=ddeeffe4db39448f8f8bc6279c70bfae, role_id=0487ef7e943e4701915cb8ce0f2899d7)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:35:26.451628 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ffb4501-b0bf-4fd8-8284-97a7ad2d1e49 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:35:26.471241 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1253/2493] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 01:35:26 2022] PUT /identity/v3/projects/d22b44e41e7c46ba9e2e32fff048ade1/users/ddeeffe4db39448f8f8bc6279c70bfae/roles/0487ef7e943e4701915cb8ce0f2899d7 => generated 0 bytes in 64 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:35:26.486112 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fbfc2a3-9f8b-4a4c-85c3-c3a0881376b4 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:26.486387 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fbfc2a3-9f8b-4a4c-85c3-c3a0881376b4 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:26.486658 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fbfc2a3-9f8b-4a4c-85c3-c3a0881376b4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:26.515265 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-8fbfc2a3-9f8b-4a4c-85c3-c3a0881376b4 None None] MFA Rules not processed for user `ddeeffe4db39448f8f8bc6279c70bfae`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:35:26.516621 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-8fbfc2a3-9f8b-4a4c-85c3-c3a0881376b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:35:26.598479 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1241/2494] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:35:26 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:35:26.603531 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ffb4501-b0bf-4fd8-8284-97a7ad2d1e49 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:35:26.673706 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ffb4501-b0bf-4fd8-8284-97a7ad2d1e49 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:35:26.680195 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3941889-e68e-4cb5-b6a2-f0c7437638f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:35:26.680803 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3941889-e68e-4cb5-b6a2-f0c7437638f7 admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:26.681466 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3941889-e68e-4cb5-b6a2-f0c7437638f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:26.681738 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3941889-e68e-4cb5-b6a2-f0c7437638f7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:26.682368 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3941889-e68e-4cb5-b6a2-f0c7437638f7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:35:26.684295 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3941889-e68e-4cb5-b6a2-f0c7437638f7 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:35:26.717497 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1254/2495] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 01:35:26 2022] POST /identity/v3/projects => generated 387 bytes in 115 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 01:35:26.721905 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fbfc2a3-9f8b-4a4c-85c3-c3a0881376b4 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:35:26.734522 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fbfc2a3-9f8b-4a4c-85c3-c3a0881376b4 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:35:26.800290 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b6bdb9c-b695-4035-a877-a6ff82d51e20 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:35:26.800930 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b6bdb9c-b695-4035-a877-a6ff82d51e20 admin admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:26.801238 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b6bdb9c-b695-4035-a877-a6ff82d51e20 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:26.801536 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b6bdb9c-b695-4035-a877-a6ff82d51e20 admin admin] PATH_INFO: `/v3/users` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:26.802135 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b6bdb9c-b695-4035-a877-a6ff82d51e20 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:35:26.804094 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b6bdb9c-b695-4035-a877-a6ff82d51e20 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:35:26.829948 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1242/2496] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 01:35:26 2022] POST /identity/v3/users => generated 294 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 01:35:26.834593 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3941889-e68e-4cb5-b6a2-f0c7437638f7 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:35:26.845599 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3941889-e68e-4cb5-b6a2-f0c7437638f7 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:35:26.850711 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a57786c-571a-4b3c-aa35-7f153cda5681 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:35:26.851310 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a57786c-571a-4b3c-aa35-7f153cda5681 admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:26.851586 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a57786c-571a-4b3c-aa35-7f153cda5681 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:26.851918 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a57786c-571a-4b3c-aa35-7f153cda5681 admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:26.852522 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a57786c-571a-4b3c-aa35-7f153cda5681 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:35:26.854160 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a57786c-571a-4b3c-aa35-7f153cda5681 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:35:26.865058 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1255/2497] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 01:35:26 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 01:35:26.869246 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b6bdb9c-b695-4035-a877-a6ff82d51e20 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:35:26.879718 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b6bdb9c-b695-4035-a877-a6ff82d51e20 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:35:26.885321 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f42e0355-b815-4145-a360-8b79a75b19cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:35:26.885932 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f42e0355-b815-4145-a360-8b79a75b19cf admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:26.886193 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f42e0355-b815-4145-a360-8b79a75b19cf admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:26.886460 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f42e0355-b815-4145-a360-8b79a75b19cf admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:26.887103 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f42e0355-b815-4145-a360-8b79a75b19cf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:35:26.888722 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f42e0355-b815-4145-a360-8b79a75b19cf admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:35:26.899486 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1243/2498] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 01:35:26 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 01:35:26.903898 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a57786c-571a-4b3c-aa35-7f153cda5681 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:35:26.914133 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a57786c-571a-4b3c-aa35-7f153cda5681 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:35:26.918899 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66ad577a-4c10-4847-9e0f-966783f9dde3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:35:26.919539 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ad577a-4c10-4847-9e0f-966783f9dde3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:26.919818 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ad577a-4c10-4847-9e0f-966783f9dde3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:26.920086 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ad577a-4c10-4847-9e0f-966783f9dde3 admin admin] PATH_INFO: `/v3/projects/aca5ca74febb4a418af5baaf2d3914b9/users/ea69a632f0824e2b9ab3723ac68a875c/roles/0487ef7e943e4701915cb8ce0f2899d7` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:26.935236 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66ad577a-4c10-4847-9e0f-966783f9dde3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=aca5ca74febb4a418af5baaf2d3914b9, user_id=ea69a632f0824e2b9ab3723ac68a875c, role_id=0487ef7e943e4701915cb8ce0f2899d7)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:35:26.937846 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66ad577a-4c10-4847-9e0f-966783f9dde3 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:35:26.956636 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1256/2499] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 01:35:26 2022] PUT /identity/v3/projects/aca5ca74febb4a418af5baaf2d3914b9/users/ea69a632f0824e2b9ab3723ac68a875c/roles/0487ef7e943e4701915cb8ce0f2899d7 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:35:26.967112 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39821b36-c338-4caf-a91f-704d5e373873 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:26.967436 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39821b36-c338-4caf-a91f-704d5e373873 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:26.967709 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39821b36-c338-4caf-a91f-704d5e373873 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:27.004787 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-39821b36-c338-4caf-a91f-704d5e373873 None None] MFA Rules not processed for user `ea69a632f0824e2b9ab3723ac68a875c`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:35:27.006093 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-39821b36-c338-4caf-a91f-704d5e373873 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:35:27.010609 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:35:27.075104 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1244/2500] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:35:26 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:35:27.080337 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66ad577a-4c10-4847-9e0f-966783f9dde3 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:35:27.153376 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66ad577a-4c10-4847-9e0f-966783f9dde3 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:35:27.158580 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0980b37e-d904-49ab-a5cb-4c94ada833a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:35:27.159281 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0980b37e-d904-49ab-a5cb-4c94ada833a9 admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:27.159650 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0980b37e-d904-49ab-a5cb-4c94ada833a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:27.159950 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0980b37e-d904-49ab-a5cb-4c94ada833a9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:27.160701 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0980b37e-d904-49ab-a5cb-4c94ada833a9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:35:27.162670 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0980b37e-d904-49ab-a5cb-4c94ada833a9 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:35:27.176549 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:35:27.205773 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1257/2501] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 01:35:27 2022] POST /identity/v3/projects => generated 385 bytes in 123 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 01:35:27.207215 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39821b36-c338-4caf-a91f-704d5e373873 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:35:27.218776 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39821b36-c338-4caf-a91f-704d5e373873 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:35:27.276288 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74620935-83be-4be3-8406-0f232d7313d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:35:27.276847 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74620935-83be-4be3-8406-0f232d7313d1 admin admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:27.277145 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74620935-83be-4be3-8406-0f232d7313d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:27.277431 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74620935-83be-4be3-8406-0f232d7313d1 admin admin] PATH_INFO: `/v3/users` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:27.278056 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74620935-83be-4be3-8406-0f232d7313d1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:35:27.280027 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74620935-83be-4be3-8406-0f232d7313d1 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:35:27.303625 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1245/2502] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 01:35:27 2022] POST /identity/v3/users => generated 299 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 01:35:27.308100 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0980b37e-d904-49ab-a5cb-4c94ada833a9 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:35:27.318409 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0980b37e-d904-49ab-a5cb-4c94ada833a9 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:35:27.322966 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84846339-8155-4f92-96db-e356e08d1c6f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:35:27.323568 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84846339-8155-4f92-96db-e356e08d1c6f admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:27.323846 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84846339-8155-4f92-96db-e356e08d1c6f admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:27.324115 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84846339-8155-4f92-96db-e356e08d1c6f admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:27.324754 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84846339-8155-4f92-96db-e356e08d1c6f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:35:27.326408 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84846339-8155-4f92-96db-e356e08d1c6f admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:35:27.336888 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1258/2503] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 01:35:27 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 01:35:27.340948 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74620935-83be-4be3-8406-0f232d7313d1 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:35:27.351322 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74620935-83be-4be3-8406-0f232d7313d1 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:35:27.357277 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c2f9bae-4ea8-42f3-a119-647905bf2b84 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:35:27.358324 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c2f9bae-4ea8-42f3-a119-647905bf2b84 admin admin] REQUEST_METHOD: `PUT` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:27.358600 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c2f9bae-4ea8-42f3-a119-647905bf2b84 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:27.358923 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c2f9bae-4ea8-42f3-a119-647905bf2b84 admin admin] PATH_INFO: `/v3/projects/771780f969144629a23b3180adabdcdd/users/6449d59d90d349fd82b3df34134344d5/roles/f08d6043261747e3b4763c744eef7dfb` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:27.375248 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c2f9bae-4ea8-42f3-a119-647905bf2b84 admin admin] RBAC: Authorizing `identity:create_grant(project_id=771780f969144629a23b3180adabdcdd, user_id=6449d59d90d349fd82b3df34134344d5, role_id=f08d6043261747e3b4763c744eef7dfb)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:35:27.378085 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c2f9bae-4ea8-42f3-a119-647905bf2b84 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:35:27.392625 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1246/2504] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 01:35:27 2022] PUT /identity/v3/projects/771780f969144629a23b3180adabdcdd/users/6449d59d90d349fd82b3df34134344d5/roles/f08d6043261747e3b4763c744eef7dfb => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 28 01:35:27.396771 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84846339-8155-4f92-96db-e356e08d1c6f admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:35:27.407150 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84846339-8155-4f92-96db-e356e08d1c6f admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:35:27.482158 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-318b4559-3945-477d-8564-0997a4c94a8a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:35:27.482789 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-318b4559-3945-477d-8564-0997a4c94a8a admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:27.483070 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-318b4559-3945-477d-8564-0997a4c94a8a admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:27.483341 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-318b4559-3945-477d-8564-0997a4c94a8a admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:27.484149 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-318b4559-3945-477d-8564-0997a4c94a8a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:35:27.486785 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-318b4559-3945-477d-8564-0997a4c94a8a admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:35:27.499260 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1259/2505] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 01:35:27 2022] GET /identity/v3/roles => generated 1420 bytes in 103 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 28 01:35:27.503919 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c2f9bae-4ea8-42f3-a119-647905bf2b84 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:35:27.514671 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c2f9bae-4ea8-42f3-a119-647905bf2b84 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:35:27.520283 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c405aedd-af75-4874-97b6-0493de381dfe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:35:27.521136 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c405aedd-af75-4874-97b6-0493de381dfe admin admin] REQUEST_METHOD: `PUT` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:27.521507 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c405aedd-af75-4874-97b6-0493de381dfe admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:27.521835 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c405aedd-af75-4874-97b6-0493de381dfe admin admin] PATH_INFO: `/v3/projects/771780f969144629a23b3180adabdcdd/users/6449d59d90d349fd82b3df34134344d5/roles/f08d6043261747e3b4763c744eef7dfb` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:27.524049 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c405aedd-af75-4874-97b6-0493de381dfe admin admin] RBAC: Authorizing `identity:create_grant(project_id=771780f969144629a23b3180adabdcdd, user_id=6449d59d90d349fd82b3df34134344d5, role_id=f08d6043261747e3b4763c744eef7dfb)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:35:27.527472 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c405aedd-af75-4874-97b6-0493de381dfe admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:35:27.540893 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1247/2506] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 01:35:27 2022] PUT /identity/v3/projects/771780f969144629a23b3180adabdcdd/users/6449d59d90d349fd82b3df34134344d5/roles/f08d6043261747e3b4763c744eef7dfb => generated 0 bytes in 38 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:35:27.554147 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f472ec63-9eb6-406e-bc6b-f4a7c3d518c9 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:27.554437 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f472ec63-9eb6-406e-bc6b-f4a7c3d518c9 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:27.554792 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f472ec63-9eb6-406e-bc6b-f4a7c3d518c9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:27.588731 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-f472ec63-9eb6-406e-bc6b-f4a7c3d518c9 None None] MFA Rules not processed for user `6449d59d90d349fd82b3df34134344d5`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:35:27.590163 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-f472ec63-9eb6-406e-bc6b-f4a7c3d518c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:35:27.667150 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1260/2507] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:35:27 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:35:27.678614 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c405aedd-af75-4874-97b6-0493de381dfe admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:35:27.680569 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-c405aedd-af75-4874-97b6-0493de381dfe admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:35:27.750842 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c405aedd-af75-4874-97b6-0493de381dfe admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:35:27.754757 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd04fefa-96a8-4428-8dfb-e488b512da93 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:35:27.755389 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd04fefa-96a8-4428-8dfb-e488b512da93 service cinder] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:27.755662 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd04fefa-96a8-4428-8dfb-e488b512da93 service cinder] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:27.755924 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd04fefa-96a8-4428-8dfb-e488b512da93 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:27.765958 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd04fefa-96a8-4428-8dfb-e488b512da93 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:35:27.767547 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd04fefa-96a8-4428-8dfb-e488b512da93 service cinder] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:35:27.773333 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:35:27.779100 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1248/2508] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:35:27 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:35:28.344486 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f472ec63-9eb6-406e-bc6b-f4a7c3d518c9 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:35:28.359600 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f472ec63-9eb6-406e-bc6b-f4a7c3d518c9 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:35:28.365679 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da24a6a6-765d-41a8-abbd-75cf86bee6d6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:35:28.366314 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da24a6a6-765d-41a8-abbd-75cf86bee6d6 service cinder] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:28.366666 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da24a6a6-765d-41a8-abbd-75cf86bee6d6 service cinder] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:28.367009 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da24a6a6-765d-41a8-abbd-75cf86bee6d6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:28.430535 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da24a6a6-765d-41a8-abbd-75cf86bee6d6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:35:28.432336 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da24a6a6-765d-41a8-abbd-75cf86bee6d6 service cinder] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:35:28.435945 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:35:28.442416 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1261/2509] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:35:28 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:35:32.536933 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd04fefa-96a8-4428-8dfb-e488b512da93 service cinder] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:35:32.547290 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd04fefa-96a8-4428-8dfb-e488b512da93 service cinder] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:35:32.551575 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcc17bea-de13-4e37-ab40-75e4f1b926e9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:35:32.552149 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcc17bea-de13-4e37-ab40-75e4f1b926e9 service cinder] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:32.552544 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcc17bea-de13-4e37-ab40-75e4f1b926e9 service cinder] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:32.552683 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcc17bea-de13-4e37-ab40-75e4f1b926e9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:32.615692 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcc17bea-de13-4e37-ab40-75e4f1b926e9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:35:32.617189 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcc17bea-de13-4e37-ab40-75e4f1b926e9 service cinder] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:35:32.626401 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1249/2510] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:35:32 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:35:34.849436 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da24a6a6-765d-41a8-abbd-75cf86bee6d6 service cinder] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:35:34.917184 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da24a6a6-765d-41a8-abbd-75cf86bee6d6 service cinder] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:35:34.922642 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-438c0eba-850c-466c-80ae-29e787d0f747 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:35:34.923406 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-438c0eba-850c-466c-80ae-29e787d0f747 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:34.923690 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-438c0eba-850c-466c-80ae-29e787d0f747 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:34.923964 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-438c0eba-850c-466c-80ae-29e787d0f747 admin admin] PATH_INFO: `/v3/users/ddeeffe4db39448f8f8bc6279c70bfae` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:34.925168 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-438c0eba-850c-466c-80ae-29e787d0f747 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ddeeffe4db39448f8f8bc6279c70bfae)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:35:34.927138 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-438c0eba-850c-466c-80ae-29e787d0f747 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:35:34.977490 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-438c0eba-850c-466c-80ae-29e787d0f747 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ddeeffe4db39448f8f8bc6279c70bfae'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:35:34.978221 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-438c0eba-850c-466c-80ae-29e787d0f747 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ddeeffe4db39448f8f8bc6279c70bfae'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:35:34.987684 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-438c0eba-850c-466c-80ae-29e787d0f747 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ddeeffe4db39448f8f8bc6279c70bfae'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:35:34.995729 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-438c0eba-850c-466c-80ae-29e787d0f747 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ddeeffe4db39448f8f8bc6279c70bfae'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:35:34.996683 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-438c0eba-850c-466c-80ae-29e787d0f747 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ddeeffe4db39448f8f8bc6279c70bfae'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:35:35.015596 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1262/2511] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 01:35:34 2022] DELETE /identity/v3/users/ddeeffe4db39448f8f8bc6279c70bfae => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:35:35.020267 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83a4cc4f-aee9-475c-8ada-6a0cd2f0904d None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:35.020734 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83a4cc4f-aee9-475c-8ada-6a0cd2f0904d None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:35.020845 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83a4cc4f-aee9-475c-8ada-6a0cd2f0904d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:35.040413 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-83a4cc4f-aee9-475c-8ada-6a0cd2f0904d None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:35:35.041666 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-83a4cc4f-aee9-475c-8ada-6a0cd2f0904d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:35:35.107100 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1250/2512] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:35:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:35:35.120783 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-438c0eba-850c-466c-80ae-29e787d0f747 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:35:35.122853 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-438c0eba-850c-466c-80ae-29e787d0f747 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:35:35.183586 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-438c0eba-850c-466c-80ae-29e787d0f747 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:35:35.187266 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b654f98e-7ba0-4efa-8727-fdbeed3ae537 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:35:35.187878 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b654f98e-7ba0-4efa-8727-fdbeed3ae537 service neutron] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:35.188153 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b654f98e-7ba0-4efa-8727-fdbeed3ae537 service neutron] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:35.188514 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b654f98e-7ba0-4efa-8727-fdbeed3ae537 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:35.197953 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b654f98e-7ba0-4efa-8727-fdbeed3ae537 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:35:35.199399 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b654f98e-7ba0-4efa-8727-fdbeed3ae537 service neutron] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:35:35.208999 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1263/2513] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:35:35 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:35:35.492048 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83a4cc4f-aee9-475c-8ada-6a0cd2f0904d None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:35:35.494243 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-83a4cc4f-aee9-475c-8ada-6a0cd2f0904d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:35:35.509788 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83a4cc4f-aee9-475c-8ada-6a0cd2f0904d None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:35:35.515732 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db1aac1d-0e52-4806-8979-b501f6c33295 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:35:35.516523 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db1aac1d-0e52-4806-8979-b501f6c33295 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:35.516867 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db1aac1d-0e52-4806-8979-b501f6c33295 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:35.517200 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db1aac1d-0e52-4806-8979-b501f6c33295 admin admin] PATH_INFO: `/v3/projects/d22b44e41e7c46ba9e2e32fff048ade1` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:35.518456 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db1aac1d-0e52-4806-8979-b501f6c33295 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d22b44e41e7c46ba9e2e32fff048ade1)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:35:35.520435 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db1aac1d-0e52-4806-8979-b501f6c33295 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:35:35.580121 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-db1aac1d-0e52-4806-8979-b501f6c33295 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd22b44e41e7c46ba9e2e32fff048ade1', 'request_id': 'req-db1aac1d-0e52-4806-8979-b501f6c33295'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:35:35.586050 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-db1aac1d-0e52-4806-8979-b501f6c33295 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd22b44e41e7c46ba9e2e32fff048ade1', 'request_id': 'req-db1aac1d-0e52-4806-8979-b501f6c33295'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:35:35.592397 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-db1aac1d-0e52-4806-8979-b501f6c33295 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd22b44e41e7c46ba9e2e32fff048ade1', 'request_id': 'req-db1aac1d-0e52-4806-8979-b501f6c33295'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:35:35.609381 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-db1aac1d-0e52-4806-8979-b501f6c33295 admin admin] The token cache is being invalidate because project d22b44e41e7c46ba9e2e32fff048ade1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81507) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 01:35:35.609643 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-db1aac1d-0e52-4806-8979-b501f6c33295 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:35:35.610280 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-db1aac1d-0e52-4806-8979-b501f6c33295 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:35:35.612039 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1251/2514] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 01:35:35 2022] DELETE /identity/v3/projects/d22b44e41e7c46ba9e2e32fff048ade1 => generated 0 bytes in 120 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 28 01:35:35.616329 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b654f98e-7ba0-4efa-8727-fdbeed3ae537 service neutron] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:35:35.628455 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b654f98e-7ba0-4efa-8727-fdbeed3ae537 service neutron] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:35:35.646251 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:35:35.699424 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e34c049-9cfc-4a0b-975b-936d73d855ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:35:35.700005 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e34c049-9cfc-4a0b-975b-936d73d855ca admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:35.700283 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e34c049-9cfc-4a0b-975b-936d73d855ca admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:35.700551 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e34c049-9cfc-4a0b-975b-936d73d855ca admin admin] PATH_INFO: `/v3/users/ea69a632f0824e2b9ab3723ac68a875c` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:35.701726 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e34c049-9cfc-4a0b-975b-936d73d855ca admin admin] RBAC: Authorizing `identity:delete_user(user_id=ea69a632f0824e2b9ab3723ac68a875c)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:35:35.703680 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e34c049-9cfc-4a0b-975b-936d73d855ca admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:35:35.760439 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-0e34c049-9cfc-4a0b-975b-936d73d855ca admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ea69a632f0824e2b9ab3723ac68a875c'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:35:35.761236 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-0e34c049-9cfc-4a0b-975b-936d73d855ca admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ea69a632f0824e2b9ab3723ac68a875c'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:35:35.770488 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-0e34c049-9cfc-4a0b-975b-936d73d855ca admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ea69a632f0824e2b9ab3723ac68a875c'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:35:35.777485 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-0e34c049-9cfc-4a0b-975b-936d73d855ca admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ea69a632f0824e2b9ab3723ac68a875c'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:35:35.778118 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-0e34c049-9cfc-4a0b-975b-936d73d855ca admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ea69a632f0824e2b9ab3723ac68a875c'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:35:35.795208 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1264/2515] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 01:35:35 2022] DELETE /identity/v3/users/ea69a632f0824e2b9ab3723ac68a875c => generated 0 bytes in 180 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:35:36.104048 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db1aac1d-0e52-4806-8979-b501f6c33295 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:35:36.105807 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-db1aac1d-0e52-4806-8979-b501f6c33295 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:35:36.125708 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db1aac1d-0e52-4806-8979-b501f6c33295 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:35:36.150787 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:35:36.200098 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c45bccf2-a439-41a1-a60e-b198344545a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:35:36.200744 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c45bccf2-a439-41a1-a60e-b198344545a6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:36.201057 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c45bccf2-a439-41a1-a60e-b198344545a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:36.201332 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c45bccf2-a439-41a1-a60e-b198344545a6 admin admin] PATH_INFO: `/v3/projects/aca5ca74febb4a418af5baaf2d3914b9` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:36.202475 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c45bccf2-a439-41a1-a60e-b198344545a6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=aca5ca74febb4a418af5baaf2d3914b9)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:35:36.204520 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c45bccf2-a439-41a1-a60e-b198344545a6 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:35:36.254380 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-c45bccf2-a439-41a1-a60e-b198344545a6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'aca5ca74febb4a418af5baaf2d3914b9', 'request_id': 'req-c45bccf2-a439-41a1-a60e-b198344545a6'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:35:36.260048 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-c45bccf2-a439-41a1-a60e-b198344545a6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'aca5ca74febb4a418af5baaf2d3914b9', 'request_id': 'req-c45bccf2-a439-41a1-a60e-b198344545a6'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:35:36.266244 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-c45bccf2-a439-41a1-a60e-b198344545a6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'aca5ca74febb4a418af5baaf2d3914b9', 'request_id': 'req-c45bccf2-a439-41a1-a60e-b198344545a6'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:35:36.280991 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-c45bccf2-a439-41a1-a60e-b198344545a6 admin admin] The token cache is being invalidate because project aca5ca74febb4a418af5baaf2d3914b9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81507) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 01:35:36.281340 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-c45bccf2-a439-41a1-a60e-b198344545a6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:35:36.282094 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-c45bccf2-a439-41a1-a60e-b198344545a6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:35:36.283688 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1252/2516] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 01:35:36 2022] DELETE /identity/v3/projects/aca5ca74febb4a418af5baaf2d3914b9 => generated 0 bytes in 180 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:35:36.287815 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e34c049-9cfc-4a0b-975b-936d73d855ca admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:35:36.289312 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-0e34c049-9cfc-4a0b-975b-936d73d855ca admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:35:36.302011 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e34c049-9cfc-4a0b-975b-936d73d855ca admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:35:36.359857 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0e8ac60-7f65-4ebb-b576-701946393379 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:35:36.360484 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0e8ac60-7f65-4ebb-b576-701946393379 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:36.360754 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0e8ac60-7f65-4ebb-b576-701946393379 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:36.361041 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0e8ac60-7f65-4ebb-b576-701946393379 admin admin] PATH_INFO: `/v3/users/6449d59d90d349fd82b3df34134344d5` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:36.362189 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0e8ac60-7f65-4ebb-b576-701946393379 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6449d59d90d349fd82b3df34134344d5)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:35:36.364119 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0e8ac60-7f65-4ebb-b576-701946393379 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:35:36.412935 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=83,keystone:DELETE=16,keystone:INSERT=1 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:35:36.419186 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-f0e8ac60-7f65-4ebb-b576-701946393379 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6449d59d90d349fd82b3df34134344d5'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:35:36.420269 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-f0e8ac60-7f65-4ebb-b576-701946393379 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6449d59d90d349fd82b3df34134344d5'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:35:36.430346 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-f0e8ac60-7f65-4ebb-b576-701946393379 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6449d59d90d349fd82b3df34134344d5'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:35:36.437811 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-f0e8ac60-7f65-4ebb-b576-701946393379 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6449d59d90d349fd82b3df34134344d5'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:35:36.438596 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-f0e8ac60-7f65-4ebb-b576-701946393379 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6449d59d90d349fd82b3df34134344d5'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:35:36.456748 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1265/2517] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 01:35:36 2022] DELETE /identity/v3/users/6449d59d90d349fd82b3df34134344d5 => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:35:36.745835 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c45bccf2-a439-41a1-a60e-b198344545a6 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:35:36.747381 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-c45bccf2-a439-41a1-a60e-b198344545a6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:35:36.761274 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c45bccf2-a439-41a1-a60e-b198344545a6 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:35:36.833677 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edbce642-5d53-4f0a-91ef-8ae3ac04076f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:35:36.834795 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edbce642-5d53-4f0a-91ef-8ae3ac04076f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:36.835306 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edbce642-5d53-4f0a-91ef-8ae3ac04076f admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:36.835802 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edbce642-5d53-4f0a-91ef-8ae3ac04076f admin admin] PATH_INFO: `/v3/projects/771780f969144629a23b3180adabdcdd` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:36.837860 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edbce642-5d53-4f0a-91ef-8ae3ac04076f admin admin] RBAC: Authorizing `identity:delete_project(project_id=771780f969144629a23b3180adabdcdd)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:35:36.840739 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edbce642-5d53-4f0a-91ef-8ae3ac04076f admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:35:36.903345 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-edbce642-5d53-4f0a-91ef-8ae3ac04076f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '771780f969144629a23b3180adabdcdd', 'request_id': 'req-edbce642-5d53-4f0a-91ef-8ae3ac04076f'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:35:36.909605 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:35:36.912202 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-edbce642-5d53-4f0a-91ef-8ae3ac04076f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '771780f969144629a23b3180adabdcdd', 'request_id': 'req-edbce642-5d53-4f0a-91ef-8ae3ac04076f'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:35:36.924478 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-edbce642-5d53-4f0a-91ef-8ae3ac04076f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '771780f969144629a23b3180adabdcdd', 'request_id': 'req-edbce642-5d53-4f0a-91ef-8ae3ac04076f'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:35:36.939815 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-edbce642-5d53-4f0a-91ef-8ae3ac04076f admin admin] The token cache is being invalidate because project 771780f969144629a23b3180adabdcdd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81507) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 01:35:36.940108 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-edbce642-5d53-4f0a-91ef-8ae3ac04076f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:35:36.940735 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-edbce642-5d53-4f0a-91ef-8ae3ac04076f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:35:36.942383 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1253/2518] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 01:35:36 2022] DELETE /identity/v3/projects/771780f969144629a23b3180adabdcdd => generated 0 bytes in 197 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:35:36.949358 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf6da6b5-8c5a-4cd2-8349-c61d94fa3a74 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:36.949636 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf6da6b5-8c5a-4cd2-8349-c61d94fa3a74 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:36.949907 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf6da6b5-8c5a-4cd2-8349-c61d94fa3a74 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:36.969857 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-cf6da6b5-8c5a-4cd2-8349-c61d94fa3a74 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:35:36.970997 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-cf6da6b5-8c5a-4cd2-8349-c61d94fa3a74 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:35:37.032248 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1266/2519] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:35:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:35:37.041209 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a33512f6-a559-4838-8ef3-eec9cafa1c78 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:37.041497 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a33512f6-a559-4838-8ef3-eec9cafa1c78 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:37.041774 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a33512f6-a559-4838-8ef3-eec9cafa1c78 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:37.062873 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-a33512f6-a559-4838-8ef3-eec9cafa1c78 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:35:37.064146 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-a33512f6-a559-4838-8ef3-eec9cafa1c78 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:35:37.073153 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1254/2520] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:35:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:35:37.078348 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf6da6b5-8c5a-4cd2-8349-c61d94fa3a74 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:35:37.090034 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf6da6b5-8c5a-4cd2-8349-c61d94fa3a74 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:35:37.097216 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a118317-322f-4de6-8022-2b821e99495f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:35:37.097830 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a118317-322f-4de6-8022-2b821e99495f admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:37.098109 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a118317-322f-4de6-8022-2b821e99495f admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:37.098374 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a118317-322f-4de6-8022-2b821e99495f admin admin] PATH_INFO: `/v3/domains` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:37.099026 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a118317-322f-4de6-8022-2b821e99495f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 01:35:37.099317 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a118317-322f-4de6-8022-2b821e99495f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:35:37.101364 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a118317-322f-4de6-8022-2b821e99495f admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:35:37.117899 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1267/2521] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Aug 28 01:35:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 01:35:37.119591 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a33512f6-a559-4838-8ef3-eec9cafa1c78 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:35:37.129226 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a33512f6-a559-4838-8ef3-eec9cafa1c78 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:35:37.134039 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a639f739-eb59-49a5-a3f6-39fe16a73ed1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:35:37.134641 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a639f739-eb59-49a5-a3f6-39fe16a73ed1 admin admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:37.134916 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a639f739-eb59-49a5-a3f6-39fe16a73ed1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:37.135186 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a639f739-eb59-49a5-a3f6-39fe16a73ed1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:37.135824 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a639f739-eb59-49a5-a3f6-39fe16a73ed1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:35:37.137772 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a639f739-eb59-49a5-a3f6-39fe16a73ed1 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:35:37.169289 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1255/2522] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 01:35:37 2022] POST /identity/v3/projects => generated 383 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 01:35:37.173711 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a118317-322f-4de6-8022-2b821e99495f admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:35:37.185624 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a118317-322f-4de6-8022-2b821e99495f admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:35:37.246319 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca39ef84-fa2e-4153-9218-fa85f936073c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:35:37.246871 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca39ef84-fa2e-4153-9218-fa85f936073c admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:37.247151 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca39ef84-fa2e-4153-9218-fa85f936073c admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:37.247408 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca39ef84-fa2e-4153-9218-fa85f936073c admin admin] PATH_INFO: `/v3/users` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:37.248034 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca39ef84-fa2e-4153-9218-fa85f936073c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:35:37.250046 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca39ef84-fa2e-4153-9218-fa85f936073c admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:35:37.275959 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1268/2523] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 01:35:37 2022] POST /identity/v3/users => generated 292 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 01:35:37.280235 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a639f739-eb59-49a5-a3f6-39fe16a73ed1 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:35:37.291001 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a639f739-eb59-49a5-a3f6-39fe16a73ed1 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:35:37.297476 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14663a32-e171-418b-94f9-4ada6ae18f60 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:35:37.298087 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14663a32-e171-418b-94f9-4ada6ae18f60 admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:37.298359 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14663a32-e171-418b-94f9-4ada6ae18f60 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:37.298691 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14663a32-e171-418b-94f9-4ada6ae18f60 admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:37.299338 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14663a32-e171-418b-94f9-4ada6ae18f60 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:35:37.300991 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14663a32-e171-418b-94f9-4ada6ae18f60 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:35:37.312948 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1256/2524] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 01:35:37 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 01:35:37.317914 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca39ef84-fa2e-4153-9218-fa85f936073c admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:35:37.333089 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca39ef84-fa2e-4153-9218-fa85f936073c admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:35:37.341063 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-831e7f6f-5767-4f22-9871-fe976c15ef64 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:35:37.341915 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-831e7f6f-5767-4f22-9871-fe976c15ef64 admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:37.342262 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-831e7f6f-5767-4f22-9871-fe976c15ef64 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:37.342602 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-831e7f6f-5767-4f22-9871-fe976c15ef64 admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:37.343580 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-831e7f6f-5767-4f22-9871-fe976c15ef64 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:35:37.346091 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-831e7f6f-5767-4f22-9871-fe976c15ef64 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:35:37.356697 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1269/2525] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 01:35:37 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 01:35:37.362422 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14663a32-e171-418b-94f9-4ada6ae18f60 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:35:37.372870 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14663a32-e171-418b-94f9-4ada6ae18f60 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:35:37.377954 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d474c4d-e7de-4e20-81ad-a3a94f9121ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:35:37.378607 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d474c4d-e7de-4e20-81ad-a3a94f9121ef admin admin] REQUEST_METHOD: `PUT` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:37.378883 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d474c4d-e7de-4e20-81ad-a3a94f9121ef admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:37.379150 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d474c4d-e7de-4e20-81ad-a3a94f9121ef admin admin] PATH_INFO: `/v3/projects/5fa8a4f2f4f948158f226337e6e34706/users/b0a320fd0e6a4d1f86492bb7d444ac65/roles/0487ef7e943e4701915cb8ce0f2899d7` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:37.395263 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d474c4d-e7de-4e20-81ad-a3a94f9121ef admin admin] RBAC: Authorizing `identity:create_grant(project_id=5fa8a4f2f4f948158f226337e6e34706, user_id=b0a320fd0e6a4d1f86492bb7d444ac65, role_id=0487ef7e943e4701915cb8ce0f2899d7)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:35:37.397987 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d474c4d-e7de-4e20-81ad-a3a94f9121ef admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:35:37.414310 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1257/2526] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 01:35:37 2022] PUT /identity/v3/projects/5fa8a4f2f4f948158f226337e6e34706/users/b0a320fd0e6a4d1f86492bb7d444ac65/roles/0487ef7e943e4701915cb8ce0f2899d7 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 28 01:35:37.429739 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-027077ac-0fd0-44da-8ffd-6ab49c0763c5 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:37.430170 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-027077ac-0fd0-44da-8ffd-6ab49c0763c5 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:37.430447 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-027077ac-0fd0-44da-8ffd-6ab49c0763c5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:37.464479 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-027077ac-0fd0-44da-8ffd-6ab49c0763c5 None None] MFA Rules not processed for user `b0a320fd0e6a4d1f86492bb7d444ac65`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:35:37.465793 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-027077ac-0fd0-44da-8ffd-6ab49c0763c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:35:37.479993 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=92,keystone:DELETE=4,keystone:INSERT=4 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:35:37.552979 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1270/2527] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:35:37 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:35:37.560687 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d474c4d-e7de-4e20-81ad-a3a94f9121ef admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:35:37.636531 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d474c4d-e7de-4e20-81ad-a3a94f9121ef admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:35:37.642607 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c26e3f0-629e-42b0-a6af-c2b4ab495ae1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:35:37.643271 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c26e3f0-629e-42b0-a6af-c2b4ab495ae1 admin admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:37.643551 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c26e3f0-629e-42b0-a6af-c2b4ab495ae1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:37.643817 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c26e3f0-629e-42b0-a6af-c2b4ab495ae1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:37.644455 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c26e3f0-629e-42b0-a6af-c2b4ab495ae1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:35:37.646417 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c26e3f0-629e-42b0-a6af-c2b4ab495ae1 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:35:37.677087 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1258/2528] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 01:35:37 2022] POST /identity/v3/projects => generated 383 bytes in 117 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 01:35:37.681621 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-027077ac-0fd0-44da-8ffd-6ab49c0763c5 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:35:37.692592 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-027077ac-0fd0-44da-8ffd-6ab49c0763c5 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:35:37.752590 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bcc6dec-baba-4edb-8132-114441802bbe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:35:37.753457 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bcc6dec-baba-4edb-8132-114441802bbe admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:37.753742 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bcc6dec-baba-4edb-8132-114441802bbe admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:37.754014 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bcc6dec-baba-4edb-8132-114441802bbe admin admin] PATH_INFO: `/v3/users` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:37.754672 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bcc6dec-baba-4edb-8132-114441802bbe admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:35:37.756587 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bcc6dec-baba-4edb-8132-114441802bbe admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:35:37.779774 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1271/2529] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 01:35:37 2022] POST /identity/v3/users => generated 298 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 01:35:37.784733 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c26e3f0-629e-42b0-a6af-c2b4ab495ae1 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:35:37.800819 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c26e3f0-629e-42b0-a6af-c2b4ab495ae1 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:35:37.808044 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b775b613-e26d-4947-9c02-46544a872372 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:35:37.808608 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b775b613-e26d-4947-9c02-46544a872372 admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:37.808879 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b775b613-e26d-4947-9c02-46544a872372 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:37.809179 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b775b613-e26d-4947-9c02-46544a872372 admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:37.809809 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b775b613-e26d-4947-9c02-46544a872372 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:35:37.811487 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b775b613-e26d-4947-9c02-46544a872372 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:35:37.822449 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1259/2530] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 01:35:37 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 01:35:37.826731 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bcc6dec-baba-4edb-8132-114441802bbe admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:35:37.837006 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bcc6dec-baba-4edb-8132-114441802bbe admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:35:37.842090 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3ae8a43-b7ab-4c32-a2ee-80dbedef2532 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:35:37.842740 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3ae8a43-b7ab-4c32-a2ee-80dbedef2532 admin admin] REQUEST_METHOD: `PUT` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:37.843024 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3ae8a43-b7ab-4c32-a2ee-80dbedef2532 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:37.843291 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3ae8a43-b7ab-4c32-a2ee-80dbedef2532 admin admin] PATH_INFO: `/v3/projects/c3d5b98590734fd69156b5ffc6fe78fc/users/a3df2c8a4691451b9a55dcc00358a1fb/roles/f08d6043261747e3b4763c744eef7dfb` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:37.858476 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3ae8a43-b7ab-4c32-a2ee-80dbedef2532 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c3d5b98590734fd69156b5ffc6fe78fc, user_id=a3df2c8a4691451b9a55dcc00358a1fb, role_id=f08d6043261747e3b4763c744eef7dfb)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:35:37.861566 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3ae8a43-b7ab-4c32-a2ee-80dbedef2532 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:35:37.875999 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1272/2531] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 01:35:37 2022] PUT /identity/v3/projects/c3d5b98590734fd69156b5ffc6fe78fc/users/a3df2c8a4691451b9a55dcc00358a1fb/roles/f08d6043261747e3b4763c744eef7dfb => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:35:37.880570 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b775b613-e26d-4947-9c02-46544a872372 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:35:37.891432 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b775b613-e26d-4947-9c02-46544a872372 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:35:37.941412 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:DELETE=2,keystone:SELECT=94,keystone:INSERT=4 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:35:37.953001 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33f26459-bed4-455d-86a0-b6493a00ffa1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:35:37.954216 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33f26459-bed4-455d-86a0-b6493a00ffa1 admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:37.955086 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33f26459-bed4-455d-86a0-b6493a00ffa1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:37.955419 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33f26459-bed4-455d-86a0-b6493a00ffa1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:37.956387 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33f26459-bed4-455d-86a0-b6493a00ffa1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:35:37.958928 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33f26459-bed4-455d-86a0-b6493a00ffa1 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:35:37.972239 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1260/2532] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 01:35:37 2022] GET /identity/v3/roles => generated 1420 bytes in 92 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 28 01:35:37.976562 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3ae8a43-b7ab-4c32-a2ee-80dbedef2532 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:35:37.987829 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3ae8a43-b7ab-4c32-a2ee-80dbedef2532 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:35:37.992642 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe3f0e4a-7a90-481e-afdf-4722d73f6527 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:35:37.993355 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe3f0e4a-7a90-481e-afdf-4722d73f6527 admin admin] REQUEST_METHOD: `PUT` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:37.993632 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe3f0e4a-7a90-481e-afdf-4722d73f6527 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:37.993903 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe3f0e4a-7a90-481e-afdf-4722d73f6527 admin admin] PATH_INFO: `/v3/projects/c3d5b98590734fd69156b5ffc6fe78fc/users/a3df2c8a4691451b9a55dcc00358a1fb/roles/f08d6043261747e3b4763c744eef7dfb` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:37.995855 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe3f0e4a-7a90-481e-afdf-4722d73f6527 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c3d5b98590734fd69156b5ffc6fe78fc, user_id=a3df2c8a4691451b9a55dcc00358a1fb, role_id=f08d6043261747e3b4763c744eef7dfb)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:35:37.998506 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe3f0e4a-7a90-481e-afdf-4722d73f6527 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:35:38.009973 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1273/2533] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 01:35:37 2022] PUT /identity/v3/projects/c3d5b98590734fd69156b5ffc6fe78fc/users/a3df2c8a4691451b9a55dcc00358a1fb/roles/f08d6043261747e3b4763c744eef7dfb => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:35:38.020555 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33d1ead7-cef3-4b92-8f7e-c485626a52be None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:38.020838 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33d1ead7-cef3-4b92-8f7e-c485626a52be None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:38.021138 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33d1ead7-cef3-4b92-8f7e-c485626a52be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:38.053084 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-33d1ead7-cef3-4b92-8f7e-c485626a52be None None] MFA Rules not processed for user `a3df2c8a4691451b9a55dcc00358a1fb`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:35:38.054296 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-33d1ead7-cef3-4b92-8f7e-c485626a52be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:35:38.127562 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1261/2534] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:35:38 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:35:38.143099 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe3f0e4a-7a90-481e-afdf-4722d73f6527 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:35:38.144948 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-fe3f0e4a-7a90-481e-afdf-4722d73f6527 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:35:38.207293 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe3f0e4a-7a90-481e-afdf-4722d73f6527 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:35:38.215123 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06afa033-998e-4ba1-bc4f-8ae791d688ae service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:35:38.215382 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06afa033-998e-4ba1-bc4f-8ae791d688ae service cinder] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:38.215582 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06afa033-998e-4ba1-bc4f-8ae791d688ae service cinder] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:38.215966 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06afa033-998e-4ba1-bc4f-8ae791d688ae service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:38.231081 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06afa033-998e-4ba1-bc4f-8ae791d688ae service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:35:38.233163 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06afa033-998e-4ba1-bc4f-8ae791d688ae service cinder] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:35:38.244549 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1274/2535] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:35:38 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:35:38.365682 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33d1ead7-cef3-4b92-8f7e-c485626a52be None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:35:38.375817 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33d1ead7-cef3-4b92-8f7e-c485626a52be None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:35:38.379464 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-735c1a54-882e-4ea9-9e27-b230043aa07f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:35:38.380066 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-735c1a54-882e-4ea9-9e27-b230043aa07f service cinder] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:38.380339 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-735c1a54-882e-4ea9-9e27-b230043aa07f service cinder] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:38.380611 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-735c1a54-882e-4ea9-9e27-b230043aa07f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:38.441818 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-735c1a54-882e-4ea9-9e27-b230043aa07f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:35:38.443265 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-735c1a54-882e-4ea9-9e27-b230043aa07f service cinder] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:35:38.452526 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1262/2536] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:35:38 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:35:41.079263 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06afa033-998e-4ba1-bc4f-8ae791d688ae service cinder] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:35:41.090198 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:35:41.153568 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06afa033-998e-4ba1-bc4f-8ae791d688ae service cinder] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:35:41.158137 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85deb884-04a2-49d3-801b-076b6e030ffb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:35:41.158894 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85deb884-04a2-49d3-801b-076b6e030ffb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:41.159178 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85deb884-04a2-49d3-801b-076b6e030ffb admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:41.159450 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85deb884-04a2-49d3-801b-076b6e030ffb admin admin] PATH_INFO: `/v3/users/b0a320fd0e6a4d1f86492bb7d444ac65` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:41.161260 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85deb884-04a2-49d3-801b-076b6e030ffb admin admin] RBAC: Authorizing `identity:delete_user(user_id=b0a320fd0e6a4d1f86492bb7d444ac65)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:35:41.163031 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85deb884-04a2-49d3-801b-076b6e030ffb admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:35:41.213349 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-85deb884-04a2-49d3-801b-076b6e030ffb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b0a320fd0e6a4d1f86492bb7d444ac65'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:35:41.214014 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-85deb884-04a2-49d3-801b-076b6e030ffb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b0a320fd0e6a4d1f86492bb7d444ac65'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:35:41.222517 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-85deb884-04a2-49d3-801b-076b6e030ffb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b0a320fd0e6a4d1f86492bb7d444ac65'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:35:41.229387 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-85deb884-04a2-49d3-801b-076b6e030ffb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b0a320fd0e6a4d1f86492bb7d444ac65'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:35:41.230010 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-85deb884-04a2-49d3-801b-076b6e030ffb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b0a320fd0e6a4d1f86492bb7d444ac65'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:35:41.246920 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1275/2537] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 01:35:41 2022] DELETE /identity/v3/users/b0a320fd0e6a4d1f86492bb7d444ac65 => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:35:41.251455 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af54710b-a9d9-47a7-b723-17eae0d2cf23 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:41.251955 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af54710b-a9d9-47a7-b723-17eae0d2cf23 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:41.252046 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af54710b-a9d9-47a7-b723-17eae0d2cf23 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:41.271120 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-af54710b-a9d9-47a7-b723-17eae0d2cf23 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:35:41.272334 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-af54710b-a9d9-47a7-b723-17eae0d2cf23 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:35:41.333420 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1263/2538] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:35:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:35:41.345323 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85deb884-04a2-49d3-801b-076b6e030ffb admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:35:41.347035 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-85deb884-04a2-49d3-801b-076b6e030ffb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:35:41.407952 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85deb884-04a2-49d3-801b-076b6e030ffb admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:35:41.412872 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22bf477c-180e-4e29-94a0-499adff3048a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:35:41.413521 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22bf477c-180e-4e29-94a0-499adff3048a service neutron] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:41.413862 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22bf477c-180e-4e29-94a0-499adff3048a service neutron] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:41.414236 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22bf477c-180e-4e29-94a0-499adff3048a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:41.426787 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22bf477c-180e-4e29-94a0-499adff3048a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:35:41.428229 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22bf477c-180e-4e29-94a0-499adff3048a service neutron] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:35:41.438462 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1276/2539] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:35:41 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:35:41.723732 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af54710b-a9d9-47a7-b723-17eae0d2cf23 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:35:41.727498 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-af54710b-a9d9-47a7-b723-17eae0d2cf23 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:35:41.740753 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af54710b-a9d9-47a7-b723-17eae0d2cf23 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:35:41.747516 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1d0ab44-1fd4-46b9-8141-6d52b6ea69b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:35:41.748160 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1d0ab44-1fd4-46b9-8141-6d52b6ea69b4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:41.748438 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1d0ab44-1fd4-46b9-8141-6d52b6ea69b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:41.748709 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1d0ab44-1fd4-46b9-8141-6d52b6ea69b4 admin admin] PATH_INFO: `/v3/projects/5fa8a4f2f4f948158f226337e6e34706` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:41.750294 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1d0ab44-1fd4-46b9-8141-6d52b6ea69b4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5fa8a4f2f4f948158f226337e6e34706)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:35:41.752411 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1d0ab44-1fd4-46b9-8141-6d52b6ea69b4 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:35:41.761392 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:35:41.801871 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-b1d0ab44-1fd4-46b9-8141-6d52b6ea69b4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5fa8a4f2f4f948158f226337e6e34706', 'request_id': 'req-b1d0ab44-1fd4-46b9-8141-6d52b6ea69b4'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:35:41.808179 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-b1d0ab44-1fd4-46b9-8141-6d52b6ea69b4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5fa8a4f2f4f948158f226337e6e34706', 'request_id': 'req-b1d0ab44-1fd4-46b9-8141-6d52b6ea69b4'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:35:41.817195 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-b1d0ab44-1fd4-46b9-8141-6d52b6ea69b4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5fa8a4f2f4f948158f226337e6e34706', 'request_id': 'req-b1d0ab44-1fd4-46b9-8141-6d52b6ea69b4'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:35:41.832619 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-b1d0ab44-1fd4-46b9-8141-6d52b6ea69b4 admin admin] The token cache is being invalidate because project 5fa8a4f2f4f948158f226337e6e34706 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81507) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 01:35:41.832902 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-b1d0ab44-1fd4-46b9-8141-6d52b6ea69b4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:35:41.833555 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-b1d0ab44-1fd4-46b9-8141-6d52b6ea69b4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:35:41.835066 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1264/2540] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 01:35:41 2022] DELETE /identity/v3/projects/5fa8a4f2f4f948158f226337e6e34706 => generated 0 bytes in 112 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:35:41.839171 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22bf477c-180e-4e29-94a0-499adff3048a service neutron] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:35:41.848840 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22bf477c-180e-4e29-94a0-499adff3048a service neutron] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:35:41.879351 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:35:41.910341 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39de69d2-dbfc-406f-bd85-6150bf89b947 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:35:41.910896 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39de69d2-dbfc-406f-bd85-6150bf89b947 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:41.911204 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39de69d2-dbfc-406f-bd85-6150bf89b947 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:41.911451 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39de69d2-dbfc-406f-bd85-6150bf89b947 admin admin] PATH_INFO: `/v3/users/a3df2c8a4691451b9a55dcc00358a1fb` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:41.912629 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39de69d2-dbfc-406f-bd85-6150bf89b947 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a3df2c8a4691451b9a55dcc00358a1fb)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:35:41.914686 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39de69d2-dbfc-406f-bd85-6150bf89b947 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:35:41.964804 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-39de69d2-dbfc-406f-bd85-6150bf89b947 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a3df2c8a4691451b9a55dcc00358a1fb'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:35:41.965496 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-39de69d2-dbfc-406f-bd85-6150bf89b947 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a3df2c8a4691451b9a55dcc00358a1fb'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:35:41.975200 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-39de69d2-dbfc-406f-bd85-6150bf89b947 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a3df2c8a4691451b9a55dcc00358a1fb'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:35:41.983105 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-39de69d2-dbfc-406f-bd85-6150bf89b947 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a3df2c8a4691451b9a55dcc00358a1fb'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:35:41.983894 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-39de69d2-dbfc-406f-bd85-6150bf89b947 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a3df2c8a4691451b9a55dcc00358a1fb'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:35:42.001514 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1277/2541] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 01:35:41 2022] DELETE /identity/v3/users/a3df2c8a4691451b9a55dcc00358a1fb => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:35:42.290969 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1d0ab44-1fd4-46b9-8141-6d52b6ea69b4 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:35:42.292488 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-b1d0ab44-1fd4-46b9-8141-6d52b6ea69b4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:35:42.308772 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1d0ab44-1fd4-46b9-8141-6d52b6ea69b4 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:35:42.365648 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c65177a-14fc-42d9-a5bc-7ac689e170c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:35:42.366287 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c65177a-14fc-42d9-a5bc-7ac689e170c8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:42.366566 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c65177a-14fc-42d9-a5bc-7ac689e170c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:42.366835 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c65177a-14fc-42d9-a5bc-7ac689e170c8 admin admin] PATH_INFO: `/v3/projects/c3d5b98590734fd69156b5ffc6fe78fc` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:42.367960 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c65177a-14fc-42d9-a5bc-7ac689e170c8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c3d5b98590734fd69156b5ffc6fe78fc)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:35:42.369938 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c65177a-14fc-42d9-a5bc-7ac689e170c8 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:35:42.420487 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-7c65177a-14fc-42d9-a5bc-7ac689e170c8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c3d5b98590734fd69156b5ffc6fe78fc', 'request_id': 'req-7c65177a-14fc-42d9-a5bc-7ac689e170c8'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:35:42.426176 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-7c65177a-14fc-42d9-a5bc-7ac689e170c8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c3d5b98590734fd69156b5ffc6fe78fc', 'request_id': 'req-7c65177a-14fc-42d9-a5bc-7ac689e170c8'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:35:42.432694 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-7c65177a-14fc-42d9-a5bc-7ac689e170c8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c3d5b98590734fd69156b5ffc6fe78fc', 'request_id': 'req-7c65177a-14fc-42d9-a5bc-7ac689e170c8'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:35:42.447446 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-7c65177a-14fc-42d9-a5bc-7ac689e170c8 admin admin] The token cache is being invalidate because project c3d5b98590734fd69156b5ffc6fe78fc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81507) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 01:35:42.447729 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-7c65177a-14fc-42d9-a5bc-7ac689e170c8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:35:42.448361 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-7c65177a-14fc-42d9-a5bc-7ac689e170c8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:35:42.449931 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1265/2542] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 01:35:42 2022] DELETE /identity/v3/projects/c3d5b98590734fd69156b5ffc6fe78fc => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:35:42.454944 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58013254-6708-4689-9aff-4957d24e1789 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:42.455431 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58013254-6708-4689-9aff-4957d24e1789 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:42.455521 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58013254-6708-4689-9aff-4957d24e1789 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:42.476716 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-58013254-6708-4689-9aff-4957d24e1789 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:35:42.477979 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-58013254-6708-4689-9aff-4957d24e1789 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:35:42.540517 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1278/2543] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:35:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:35:42.553178 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6ec4672-5649-41cb-997c-e2a749d3470d None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:42.553466 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6ec4672-5649-41cb-997c-e2a749d3470d None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:42.553932 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6ec4672-5649-41cb-997c-e2a749d3470d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:42.578068 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-c6ec4672-5649-41cb-997c-e2a749d3470d None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:35:42.579313 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-c6ec4672-5649-41cb-997c-e2a749d3470d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:35:42.587385 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1266/2544] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:35:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:35:42.591816 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58013254-6708-4689-9aff-4957d24e1789 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:35:42.606617 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58013254-6708-4689-9aff-4957d24e1789 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:35:42.612388 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-315fbf77-11f4-46a5-acc5-ed038ca69ccd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:35:42.613015 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-315fbf77-11f4-46a5-acc5-ed038ca69ccd admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:42.613309 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-315fbf77-11f4-46a5-acc5-ed038ca69ccd admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:42.613573 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-315fbf77-11f4-46a5-acc5-ed038ca69ccd admin admin] PATH_INFO: `/v3/domains` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:42.614215 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-315fbf77-11f4-46a5-acc5-ed038ca69ccd admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 01:35:42.614510 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-315fbf77-11f4-46a5-acc5-ed038ca69ccd admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:35:42.616135 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-315fbf77-11f4-46a5-acc5-ed038ca69ccd admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:35:42.628575 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1279/2545] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Aug 28 01:35:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 01:35:42.633281 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6ec4672-5649-41cb-997c-e2a749d3470d None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:35:42.642889 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6ec4672-5649-41cb-997c-e2a749d3470d None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:35:42.647835 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddf7cb0d-5957-4c62-b2ef-17f299e0582a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:35:42.648437 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf7cb0d-5957-4c62-b2ef-17f299e0582a admin admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:42.648711 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf7cb0d-5957-4c62-b2ef-17f299e0582a admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:42.649077 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf7cb0d-5957-4c62-b2ef-17f299e0582a admin admin] PATH_INFO: `/v3/projects` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:42.649740 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddf7cb0d-5957-4c62-b2ef-17f299e0582a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:35:42.651650 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddf7cb0d-5957-4c62-b2ef-17f299e0582a admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:35:42.675749 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=83,keystone:DELETE=14,keystone:INSERT=3 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:35:42.683630 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1267/2546] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 01:35:42 2022] POST /identity/v3/projects => generated 383 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 01:35:42.687946 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-315fbf77-11f4-46a5-acc5-ed038ca69ccd admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:35:42.697879 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-315fbf77-11f4-46a5-acc5-ed038ca69ccd admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:35:42.735806 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:35:42.763569 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98b97126-45dd-4285-95a7-fbe74ba05ae6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:35:42.764185 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98b97126-45dd-4285-95a7-fbe74ba05ae6 admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:42.764461 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98b97126-45dd-4285-95a7-fbe74ba05ae6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:42.764701 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98b97126-45dd-4285-95a7-fbe74ba05ae6 admin admin] PATH_INFO: `/v3/users` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:42.765364 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98b97126-45dd-4285-95a7-fbe74ba05ae6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:35:42.767304 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98b97126-45dd-4285-95a7-fbe74ba05ae6 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:35:42.791118 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1280/2547] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 01:35:42 2022] POST /identity/v3/users => generated 292 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 01:35:42.795564 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddf7cb0d-5957-4c62-b2ef-17f299e0582a admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:35:42.807154 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddf7cb0d-5957-4c62-b2ef-17f299e0582a admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:35:42.812298 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49f4a01e-1315-4b9c-b7f9-12ea37368c3b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:35:42.812925 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f4a01e-1315-4b9c-b7f9-12ea37368c3b admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:42.813250 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f4a01e-1315-4b9c-b7f9-12ea37368c3b admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:42.813519 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f4a01e-1315-4b9c-b7f9-12ea37368c3b admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:42.814181 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49f4a01e-1315-4b9c-b7f9-12ea37368c3b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:35:42.815803 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49f4a01e-1315-4b9c-b7f9-12ea37368c3b admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:35:42.826761 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1268/2548] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 01:35:42 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 01:35:42.831855 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98b97126-45dd-4285-95a7-fbe74ba05ae6 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:35:42.846715 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98b97126-45dd-4285-95a7-fbe74ba05ae6 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:35:42.854373 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9ddbc92-e3fc-4e03-99f9-d660cdfe6606 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:35:42.855286 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9ddbc92-e3fc-4e03-99f9-d660cdfe6606 admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:42.855671 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9ddbc92-e3fc-4e03-99f9-d660cdfe6606 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:42.855977 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9ddbc92-e3fc-4e03-99f9-d660cdfe6606 admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:42.856947 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9ddbc92-e3fc-4e03-99f9-d660cdfe6606 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:35:42.859211 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9ddbc92-e3fc-4e03-99f9-d660cdfe6606 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:35:42.870788 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1281/2549] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 01:35:42 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 01:35:42.875400 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49f4a01e-1315-4b9c-b7f9-12ea37368c3b admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:35:42.888187 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49f4a01e-1315-4b9c-b7f9-12ea37368c3b admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:35:42.891991 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74e511c3-5f90-4a6b-8abc-9e59913158b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:35:42.892837 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74e511c3-5f90-4a6b-8abc-9e59913158b3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:42.893065 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74e511c3-5f90-4a6b-8abc-9e59913158b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:42.893407 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74e511c3-5f90-4a6b-8abc-9e59913158b3 admin admin] PATH_INFO: `/v3/projects/20782890f81c4db2a696055f13cc514c/users/43c53ed1dde3438fa85f403acda17229/roles/0487ef7e943e4701915cb8ce0f2899d7` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:42.909450 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74e511c3-5f90-4a6b-8abc-9e59913158b3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=20782890f81c4db2a696055f13cc514c, user_id=43c53ed1dde3438fa85f403acda17229, role_id=0487ef7e943e4701915cb8ce0f2899d7)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:35:42.912176 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74e511c3-5f90-4a6b-8abc-9e59913158b3 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:35:42.928792 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1269/2550] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 01:35:42 2022] PUT /identity/v3/projects/20782890f81c4db2a696055f13cc514c/users/43c53ed1dde3438fa85f403acda17229/roles/0487ef7e943e4701915cb8ce0f2899d7 => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:35:42.940791 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae58b433-f23f-4b20-826a-8d76ec862c79 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:42.941119 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae58b433-f23f-4b20-826a-8d76ec862c79 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:42.941394 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae58b433-f23f-4b20-826a-8d76ec862c79 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:42.970583 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-ae58b433-f23f-4b20-826a-8d76ec862c79 None None] MFA Rules not processed for user `43c53ed1dde3438fa85f403acda17229`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:35:42.971845 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-ae58b433-f23f-4b20-826a-8d76ec862c79 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:35:43.040437 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1282/2551] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:35:42 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:35:43.049294 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74e511c3-5f90-4a6b-8abc-9e59913158b3 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:35:43.110323 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74e511c3-5f90-4a6b-8abc-9e59913158b3 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:35:43.115136 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39bcd385-0fae-4082-bd58-bf4dd7770c5b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:35:43.115772 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39bcd385-0fae-4082-bd58-bf4dd7770c5b admin admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:43.116027 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39bcd385-0fae-4082-bd58-bf4dd7770c5b admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:43.116286 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39bcd385-0fae-4082-bd58-bf4dd7770c5b admin admin] PATH_INFO: `/v3/projects` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:43.116929 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39bcd385-0fae-4082-bd58-bf4dd7770c5b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:35:43.118903 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39bcd385-0fae-4082-bd58-bf4dd7770c5b admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:35:43.151053 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1270/2552] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 01:35:43 2022] POST /identity/v3/projects => generated 385 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 01:35:43.155474 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae58b433-f23f-4b20-826a-8d76ec862c79 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:35:43.166963 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae58b433-f23f-4b20-826a-8d76ec862c79 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:35:43.225301 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2cc46ca-64f2-4335-a2f2-e68144d78c14 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:35:43.225832 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2cc46ca-64f2-4335-a2f2-e68144d78c14 admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:43.226118 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2cc46ca-64f2-4335-a2f2-e68144d78c14 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:43.226379 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2cc46ca-64f2-4335-a2f2-e68144d78c14 admin admin] PATH_INFO: `/v3/users` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:43.227013 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2cc46ca-64f2-4335-a2f2-e68144d78c14 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:35:43.228993 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2cc46ca-64f2-4335-a2f2-e68144d78c14 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:35:43.253329 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1283/2553] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 01:35:43 2022] POST /identity/v3/users => generated 299 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 01:35:43.257550 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39bcd385-0fae-4082-bd58-bf4dd7770c5b admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:35:43.268447 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39bcd385-0fae-4082-bd58-bf4dd7770c5b admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:35:43.273751 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7ac69bd-c25b-49a7-a7b5-383bafc953ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:35:43.274361 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7ac69bd-c25b-49a7-a7b5-383bafc953ad admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:43.274639 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7ac69bd-c25b-49a7-a7b5-383bafc953ad admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:43.274909 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7ac69bd-c25b-49a7-a7b5-383bafc953ad admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:43.275550 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7ac69bd-c25b-49a7-a7b5-383bafc953ad admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:35:43.277205 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7ac69bd-c25b-49a7-a7b5-383bafc953ad admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:35:43.290066 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1271/2554] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 01:35:43 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 01:35:43.294347 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2cc46ca-64f2-4335-a2f2-e68144d78c14 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:35:43.305330 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2cc46ca-64f2-4335-a2f2-e68144d78c14 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:35:43.310269 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5687e09-86bc-4094-a4e0-48d48d3659e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:35:43.310926 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5687e09-86bc-4094-a4e0-48d48d3659e0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:43.311197 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5687e09-86bc-4094-a4e0-48d48d3659e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:43.311463 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5687e09-86bc-4094-a4e0-48d48d3659e0 admin admin] PATH_INFO: `/v3/projects/367258c7483847028a4d04b6ec2780fe/users/4422925d2a5c400da30791586fa5f6a8/roles/f08d6043261747e3b4763c744eef7dfb` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:43.326455 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5687e09-86bc-4094-a4e0-48d48d3659e0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=367258c7483847028a4d04b6ec2780fe, user_id=4422925d2a5c400da30791586fa5f6a8, role_id=f08d6043261747e3b4763c744eef7dfb)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:35:43.329151 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5687e09-86bc-4094-a4e0-48d48d3659e0 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:35:43.344128 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1284/2555] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 01:35:43 2022] PUT /identity/v3/projects/367258c7483847028a4d04b6ec2780fe/users/4422925d2a5c400da30791586fa5f6a8/roles/f08d6043261747e3b4763c744eef7dfb => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:35:43.348262 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7ac69bd-c25b-49a7-a7b5-383bafc953ad admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:35:43.358315 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7ac69bd-c25b-49a7-a7b5-383bafc953ad admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:35:43.415046 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faf776d7-de45-4e7d-9d52-1576e44fae1d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:35:43.415635 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf776d7-de45-4e7d-9d52-1576e44fae1d admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:43.415904 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf776d7-de45-4e7d-9d52-1576e44fae1d admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:43.416169 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf776d7-de45-4e7d-9d52-1576e44fae1d admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:43.416789 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faf776d7-de45-4e7d-9d52-1576e44fae1d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:35:43.418416 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faf776d7-de45-4e7d-9d52-1576e44fae1d admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:35:43.428752 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1272/2556] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 01:35:43 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 01:35:43.433466 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5687e09-86bc-4094-a4e0-48d48d3659e0 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:35:43.445095 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5687e09-86bc-4094-a4e0-48d48d3659e0 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:35:43.450014 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa1d30e1-5c66-4d75-8daf-54d70ca2aae7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:35:43.450667 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa1d30e1-5c66-4d75-8daf-54d70ca2aae7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:43.450966 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa1d30e1-5c66-4d75-8daf-54d70ca2aae7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:43.451239 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa1d30e1-5c66-4d75-8daf-54d70ca2aae7 admin admin] PATH_INFO: `/v3/projects/367258c7483847028a4d04b6ec2780fe/users/4422925d2a5c400da30791586fa5f6a8/roles/f08d6043261747e3b4763c744eef7dfb` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:43.453239 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa1d30e1-5c66-4d75-8daf-54d70ca2aae7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=367258c7483847028a4d04b6ec2780fe, user_id=4422925d2a5c400da30791586fa5f6a8, role_id=f08d6043261747e3b4763c744eef7dfb)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:35:43.455949 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa1d30e1-5c66-4d75-8daf-54d70ca2aae7 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:35:43.461899 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:35:43.469565 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1285/2557] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 01:35:43 2022] PUT /identity/v3/projects/367258c7483847028a4d04b6ec2780fe/users/4422925d2a5c400da30791586fa5f6a8/roles/f08d6043261747e3b4763c744eef7dfb => generated 0 bytes in 37 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:35:43.481132 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e1064c9-ad54-461c-a65d-dd8216e7141e None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:43.481401 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e1064c9-ad54-461c-a65d-dd8216e7141e None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:43.481844 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e1064c9-ad54-461c-a65d-dd8216e7141e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:43.511340 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-2e1064c9-ad54-461c-a65d-dd8216e7141e None None] MFA Rules not processed for user `4422925d2a5c400da30791586fa5f6a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:35:43.512548 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-2e1064c9-ad54-461c-a65d-dd8216e7141e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:35:43.539240 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:35:43.589407 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1273/2558] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:35:43 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:35:43.600520 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa1d30e1-5c66-4d75-8daf-54d70ca2aae7 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:35:43.602319 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-fa1d30e1-5c66-4d75-8daf-54d70ca2aae7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:35:43.664799 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa1d30e1-5c66-4d75-8daf-54d70ca2aae7 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:35:43.669865 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbcf9f1c-6b2f-4006-887a-f54c28fe7ed5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:35:43.670476 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbcf9f1c-6b2f-4006-887a-f54c28fe7ed5 service cinder] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:43.670757 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbcf9f1c-6b2f-4006-887a-f54c28fe7ed5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:43.671017 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbcf9f1c-6b2f-4006-887a-f54c28fe7ed5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:43.727316 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbcf9f1c-6b2f-4006-887a-f54c28fe7ed5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:35:43.728780 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbcf9f1c-6b2f-4006-887a-f54c28fe7ed5 service cinder] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:35:43.738633 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1286/2559] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:35:43 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 139 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:35:48.065632 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e1064c9-ad54-461c-a65d-dd8216e7141e None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:35:48.076266 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e1064c9-ad54-461c-a65d-dd8216e7141e None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:35:48.080302 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4c54335-fada-4482-8f8d-1f9bd2fd4b80 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:35:48.080904 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4c54335-fada-4482-8f8d-1f9bd2fd4b80 service cinder] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:48.081182 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4c54335-fada-4482-8f8d-1f9bd2fd4b80 service cinder] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:48.081446 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4c54335-fada-4482-8f8d-1f9bd2fd4b80 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:48.091147 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4c54335-fada-4482-8f8d-1f9bd2fd4b80 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:35:48.092632 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4c54335-fada-4482-8f8d-1f9bd2fd4b80 service cinder] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:35:48.102701 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1274/2560] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:35:48 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:35:53.074690 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbcf9f1c-6b2f-4006-887a-f54c28fe7ed5 service cinder] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:35:53.159023 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbcf9f1c-6b2f-4006-887a-f54c28fe7ed5 service cinder] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:35:53.164607 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1082e5e-b8f8-4241-8e12-87e2c0f89f5f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:35:53.165276 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1082e5e-b8f8-4241-8e12-87e2c0f89f5f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:53.165578 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1082e5e-b8f8-4241-8e12-87e2c0f89f5f admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:53.165841 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1082e5e-b8f8-4241-8e12-87e2c0f89f5f admin admin] PATH_INFO: `/v3/users/43c53ed1dde3438fa85f403acda17229` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:53.167091 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1082e5e-b8f8-4241-8e12-87e2c0f89f5f admin admin] RBAC: Authorizing `identity:delete_user(user_id=43c53ed1dde3438fa85f403acda17229)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:35:53.169103 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1082e5e-b8f8-4241-8e12-87e2c0f89f5f admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:35:53.212481 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:35:53.222573 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-a1082e5e-b8f8-4241-8e12-87e2c0f89f5f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '43c53ed1dde3438fa85f403acda17229'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:35:53.223236 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-a1082e5e-b8f8-4241-8e12-87e2c0f89f5f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '43c53ed1dde3438fa85f403acda17229'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:35:53.231922 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-a1082e5e-b8f8-4241-8e12-87e2c0f89f5f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '43c53ed1dde3438fa85f403acda17229'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:35:53.239768 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-a1082e5e-b8f8-4241-8e12-87e2c0f89f5f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '43c53ed1dde3438fa85f403acda17229'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:35:53.240491 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-a1082e5e-b8f8-4241-8e12-87e2c0f89f5f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '43c53ed1dde3438fa85f403acda17229'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:35:53.258897 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1287/2561] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 01:35:53 2022] DELETE /identity/v3/users/43c53ed1dde3438fa85f403acda17229 => generated 0 bytes in 185 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:35:53.263636 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dec0780-5f3a-4cab-9513-0e023589ba5f None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:53.264172 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dec0780-5f3a-4cab-9513-0e023589ba5f None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:53.264172 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dec0780-5f3a-4cab-9513-0e023589ba5f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:53.284624 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-3dec0780-5f3a-4cab-9513-0e023589ba5f None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:35:53.285887 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-3dec0780-5f3a-4cab-9513-0e023589ba5f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:35:53.348327 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1275/2562] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:35:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:35:53.358496 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1082e5e-b8f8-4241-8e12-87e2c0f89f5f admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:35:53.360143 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-a1082e5e-b8f8-4241-8e12-87e2c0f89f5f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:35:53.427087 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1082e5e-b8f8-4241-8e12-87e2c0f89f5f admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:35:53.432870 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23078952-e42c-4d25-ad4c-a3c2b5b69685 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:35:53.433825 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23078952-e42c-4d25-ad4c-a3c2b5b69685 service neutron] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:53.434276 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23078952-e42c-4d25-ad4c-a3c2b5b69685 service neutron] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:53.434673 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23078952-e42c-4d25-ad4c-a3c2b5b69685 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:53.448398 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23078952-e42c-4d25-ad4c-a3c2b5b69685 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:35:53.449948 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23078952-e42c-4d25-ad4c-a3c2b5b69685 service neutron] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:35:53.459833 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1288/2563] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:35:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:35:53.574372 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dec0780-5f3a-4cab-9513-0e023589ba5f None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:35:53.575846 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-3dec0780-5f3a-4cab-9513-0e023589ba5f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:35:53.638279 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dec0780-5f3a-4cab-9513-0e023589ba5f None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:35:53.644209 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75a27855-aa6c-48c5-8fd9-1447bac809da service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:35:53.644835 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75a27855-aa6c-48c5-8fd9-1447bac809da service cinder] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:53.645136 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75a27855-aa6c-48c5-8fd9-1447bac809da service cinder] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:53.645400 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75a27855-aa6c-48c5-8fd9-1447bac809da service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:53.647323 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-75a27855-aa6c-48c5-8fd9-1447bac809da service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:35:53.708044 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:35:53.719420 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75a27855-aa6c-48c5-8fd9-1447bac809da service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:35:53.720919 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75a27855-aa6c-48c5-8fd9-1447bac809da service cinder] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:35:53.741519 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1276/2564] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:35:53 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 168 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:35:53.757451 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23078952-e42c-4d25-ad4c-a3c2b5b69685 service neutron] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:35:53.758857 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-23078952-e42c-4d25-ad4c-a3c2b5b69685 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:35:53.773993 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23078952-e42c-4d25-ad4c-a3c2b5b69685 service neutron] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:35:53.779048 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1d3981e-ae7f-44ff-bbec-4415db534047 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:35:53.779686 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1d3981e-ae7f-44ff-bbec-4415db534047 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:53.779966 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1d3981e-ae7f-44ff-bbec-4415db534047 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:53.780235 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1d3981e-ae7f-44ff-bbec-4415db534047 admin admin] PATH_INFO: `/v3/projects/20782890f81c4db2a696055f13cc514c` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:53.781519 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1d3981e-ae7f-44ff-bbec-4415db534047 admin admin] RBAC: Authorizing `identity:delete_project(project_id=20782890f81c4db2a696055f13cc514c)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:35:53.783304 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1d3981e-ae7f-44ff-bbec-4415db534047 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:35:53.834117 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-f1d3981e-ae7f-44ff-bbec-4415db534047 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '20782890f81c4db2a696055f13cc514c', 'request_id': 'req-f1d3981e-ae7f-44ff-bbec-4415db534047'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:35:53.839746 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-f1d3981e-ae7f-44ff-bbec-4415db534047 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '20782890f81c4db2a696055f13cc514c', 'request_id': 'req-f1d3981e-ae7f-44ff-bbec-4415db534047'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:35:53.846639 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-f1d3981e-ae7f-44ff-bbec-4415db534047 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '20782890f81c4db2a696055f13cc514c', 'request_id': 'req-f1d3981e-ae7f-44ff-bbec-4415db534047'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:35:53.863520 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-f1d3981e-ae7f-44ff-bbec-4415db534047 admin admin] The token cache is being invalidate because project 20782890f81c4db2a696055f13cc514c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81508) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 01:35:53.863851 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-f1d3981e-ae7f-44ff-bbec-4415db534047 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:35:53.864472 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-f1d3981e-ae7f-44ff-bbec-4415db534047 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:35:53.866141 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1289/2565] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 01:35:53 2022] DELETE /identity/v3/projects/20782890f81c4db2a696055f13cc514c => generated 0 bytes in 109 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:35:53.870104 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75a27855-aa6c-48c5-8fd9-1447bac809da service cinder] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:35:53.880913 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75a27855-aa6c-48c5-8fd9-1447bac809da service cinder] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:35:53.938171 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9a9fa39-0cb8-4f38-be4b-9acb86f38069 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:35:53.938800 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9a9fa39-0cb8-4f38-be4b-9acb86f38069 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:53.939113 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9a9fa39-0cb8-4f38-be4b-9acb86f38069 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:53.939385 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9a9fa39-0cb8-4f38-be4b-9acb86f38069 admin admin] PATH_INFO: `/v3/users/4422925d2a5c400da30791586fa5f6a8` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:53.940616 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9a9fa39-0cb8-4f38-be4b-9acb86f38069 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4422925d2a5c400da30791586fa5f6a8)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:35:53.942635 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9a9fa39-0cb8-4f38-be4b-9acb86f38069 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:35:53.999229 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-b9a9fa39-0cb8-4f38-be4b-9acb86f38069 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4422925d2a5c400da30791586fa5f6a8'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:35:54.011707 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-b9a9fa39-0cb8-4f38-be4b-9acb86f38069 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4422925d2a5c400da30791586fa5f6a8'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:35:54.022250 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-b9a9fa39-0cb8-4f38-be4b-9acb86f38069 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4422925d2a5c400da30791586fa5f6a8'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:35:54.029904 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-b9a9fa39-0cb8-4f38-be4b-9acb86f38069 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4422925d2a5c400da30791586fa5f6a8'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:35:54.030655 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-b9a9fa39-0cb8-4f38-be4b-9acb86f38069 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4422925d2a5c400da30791586fa5f6a8'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:35:54.036742 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1277/2566] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 01:35:53 2022] DELETE /identity/v3/users/4422925d2a5c400da30791586fa5f6a8 => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:35:54.332396 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1d3981e-ae7f-44ff-bbec-4415db534047 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:35:54.334250 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-f1d3981e-ae7f-44ff-bbec-4415db534047 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:35:54.352344 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1d3981e-ae7f-44ff-bbec-4415db534047 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:35:54.388733 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:35:54.426781 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5d7785f-70f6-4fe8-9b22-5340bed5e774 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:35:54.427424 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5d7785f-70f6-4fe8-9b22-5340bed5e774 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:54.427711 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5d7785f-70f6-4fe8-9b22-5340bed5e774 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:54.427963 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5d7785f-70f6-4fe8-9b22-5340bed5e774 admin admin] PATH_INFO: `/v3/projects/367258c7483847028a4d04b6ec2780fe` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:54.429132 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5d7785f-70f6-4fe8-9b22-5340bed5e774 admin admin] RBAC: Authorizing `identity:delete_project(project_id=367258c7483847028a4d04b6ec2780fe)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:35:54.431409 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5d7785f-70f6-4fe8-9b22-5340bed5e774 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:35:54.480226 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-b5d7785f-70f6-4fe8-9b22-5340bed5e774 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '367258c7483847028a4d04b6ec2780fe', 'request_id': 'req-b5d7785f-70f6-4fe8-9b22-5340bed5e774'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:35:54.486798 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-b5d7785f-70f6-4fe8-9b22-5340bed5e774 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '367258c7483847028a4d04b6ec2780fe', 'request_id': 'req-b5d7785f-70f6-4fe8-9b22-5340bed5e774'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:35:54.494731 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-b5d7785f-70f6-4fe8-9b22-5340bed5e774 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '367258c7483847028a4d04b6ec2780fe', 'request_id': 'req-b5d7785f-70f6-4fe8-9b22-5340bed5e774'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:35:54.510511 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-b5d7785f-70f6-4fe8-9b22-5340bed5e774 admin admin] The token cache is being invalidate because project 367258c7483847028a4d04b6ec2780fe was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81508) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 01:35:54.510799 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-b5d7785f-70f6-4fe8-9b22-5340bed5e774 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:35:54.511439 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-b5d7785f-70f6-4fe8-9b22-5340bed5e774 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:35:54.512976 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1290/2567] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 01:35:54 2022] DELETE /identity/v3/projects/367258c7483847028a4d04b6ec2780fe => generated 0 bytes in 182 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:35:54.519149 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-469a04f6-7acb-4540-9062-632c65a760ce None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:54.519778 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-469a04f6-7acb-4540-9062-632c65a760ce None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:54.519921 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-469a04f6-7acb-4540-9062-632c65a760ce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:54.543065 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-469a04f6-7acb-4540-9062-632c65a760ce None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:35:54.544295 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-469a04f6-7acb-4540-9062-632c65a760ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:35:54.605658 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1278/2568] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:35:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:35:54.615495 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51d81a28-68fc-4127-816e-70573ca57138 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:54.615863 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51d81a28-68fc-4127-816e-70573ca57138 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:54.616132 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51d81a28-68fc-4127-816e-70573ca57138 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:54.637275 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-51d81a28-68fc-4127-816e-70573ca57138 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:35:54.638492 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-51d81a28-68fc-4127-816e-70573ca57138 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:35:54.647963 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1291/2569] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:35:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:35:54.652418 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-469a04f6-7acb-4540-9062-632c65a760ce None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:35:54.657604 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:35:54.663973 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-469a04f6-7acb-4540-9062-632c65a760ce None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:35:54.670766 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dd60eb8-1384-44ef-8257-33010d4b7c28 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:35:54.671401 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dd60eb8-1384-44ef-8257-33010d4b7c28 admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:54.671715 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dd60eb8-1384-44ef-8257-33010d4b7c28 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:54.672013 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dd60eb8-1384-44ef-8257-33010d4b7c28 admin admin] PATH_INFO: `/v3/domains` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:54.672662 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dd60eb8-1384-44ef-8257-33010d4b7c28 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 01:35:54.672984 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dd60eb8-1384-44ef-8257-33010d4b7c28 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:35:54.674970 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dd60eb8-1384-44ef-8257-33010d4b7c28 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:35:54.688096 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1279/2570] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Aug 28 01:35:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 01:35:54.693258 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51d81a28-68fc-4127-816e-70573ca57138 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:35:54.708012 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51d81a28-68fc-4127-816e-70573ca57138 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:35:54.714608 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f33e351e-2c9b-4570-8170-44dde33ba165 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:35:54.715207 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f33e351e-2c9b-4570-8170-44dde33ba165 admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:54.715479 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f33e351e-2c9b-4570-8170-44dde33ba165 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:54.715743 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f33e351e-2c9b-4570-8170-44dde33ba165 admin admin] PATH_INFO: `/v3/projects` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:54.716360 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f33e351e-2c9b-4570-8170-44dde33ba165 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:35:54.718340 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f33e351e-2c9b-4570-8170-44dde33ba165 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:35:54.755162 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1292/2571] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 01:35:54 2022] POST /identity/v3/projects => generated 373 bytes in 63 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 01:35:54.760493 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dd60eb8-1384-44ef-8257-33010d4b7c28 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:35:54.774522 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dd60eb8-1384-44ef-8257-33010d4b7c28 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:35:54.838276 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80389275-d13c-4295-bd74-659972ec512c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:35:54.838901 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80389275-d13c-4295-bd74-659972ec512c admin admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:54.839182 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80389275-d13c-4295-bd74-659972ec512c admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:54.839453 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80389275-d13c-4295-bd74-659972ec512c admin admin] PATH_INFO: `/v3/users` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:54.840090 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80389275-d13c-4295-bd74-659972ec512c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:35:54.842074 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80389275-d13c-4295-bd74-659972ec512c admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:35:54.865533 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1280/2572] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 01:35:54 2022] POST /identity/v3/users => generated 287 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 01:35:54.870376 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f33e351e-2c9b-4570-8170-44dde33ba165 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:35:54.885003 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f33e351e-2c9b-4570-8170-44dde33ba165 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:35:54.890170 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dab126e2-3bb7-451e-a055-fe5c7614b306 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:35:54.890784 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dab126e2-3bb7-451e-a055-fe5c7614b306 admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:54.891061 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dab126e2-3bb7-451e-a055-fe5c7614b306 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:54.891334 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dab126e2-3bb7-451e-a055-fe5c7614b306 admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:54.891947 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dab126e2-3bb7-451e-a055-fe5c7614b306 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:35:54.893633 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dab126e2-3bb7-451e-a055-fe5c7614b306 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:35:54.904046 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1293/2573] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 01:35:54 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 01:35:54.908207 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80389275-d13c-4295-bd74-659972ec512c admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:35:54.918282 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80389275-d13c-4295-bd74-659972ec512c admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:35:54.923145 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d5e9578-49fa-4212-a1c8-31cb01192dcb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:35:54.923750 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d5e9578-49fa-4212-a1c8-31cb01192dcb admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:54.924029 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d5e9578-49fa-4212-a1c8-31cb01192dcb admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:54.924298 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d5e9578-49fa-4212-a1c8-31cb01192dcb admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:54.924950 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d5e9578-49fa-4212-a1c8-31cb01192dcb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:35:54.926582 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d5e9578-49fa-4212-a1c8-31cb01192dcb admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:35:54.938071 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1281/2574] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 01:35:54 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 28 01:35:54.942337 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dab126e2-3bb7-451e-a055-fe5c7614b306 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:35:54.952377 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dab126e2-3bb7-451e-a055-fe5c7614b306 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:35:54.957002 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a9d104c-96a3-47d6-804e-cc9d14039e57 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:35:54.957707 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a9d104c-96a3-47d6-804e-cc9d14039e57 admin admin] REQUEST_METHOD: `PUT` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:54.957960 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a9d104c-96a3-47d6-804e-cc9d14039e57 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:54.958229 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a9d104c-96a3-47d6-804e-cc9d14039e57 admin admin] PATH_INFO: `/v3/projects/802f856adff04f85b6731d6a97331da6/users/f757f2efd63f4526892784e78a402ccd/roles/0487ef7e943e4701915cb8ce0f2899d7` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:54.972373 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a9d104c-96a3-47d6-804e-cc9d14039e57 admin admin] RBAC: Authorizing `identity:create_grant(project_id=802f856adff04f85b6731d6a97331da6, user_id=f757f2efd63f4526892784e78a402ccd, role_id=0487ef7e943e4701915cb8ce0f2899d7)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:35:54.975045 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a9d104c-96a3-47d6-804e-cc9d14039e57 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:35:54.991151 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1294/2575] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 01:35:54 2022] PUT /identity/v3/projects/802f856adff04f85b6731d6a97331da6/users/f757f2efd63f4526892784e78a402ccd/roles/0487ef7e943e4701915cb8ce0f2899d7 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:35:55.002477 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96f4c784-9bf0-46e2-af3a-a3ea6cc278cb None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:55.002757 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96f4c784-9bf0-46e2-af3a-a3ea6cc278cb None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:55.003040 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96f4c784-9bf0-46e2-af3a-a3ea6cc278cb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:55.031447 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-96f4c784-9bf0-46e2-af3a-a3ea6cc278cb None None] MFA Rules not processed for user `f757f2efd63f4526892784e78a402ccd`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:35:55.032674 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-96f4c784-9bf0-46e2-af3a-a3ea6cc278cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:35:55.097386 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1282/2576] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:35:55 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:35:55.111519 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a9d104c-96a3-47d6-804e-cc9d14039e57 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:35:55.113494 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-5a9d104c-96a3-47d6-804e-cc9d14039e57 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:35:55.179912 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a9d104c-96a3-47d6-804e-cc9d14039e57 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:35:55.186403 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed4f0349-a335-41dc-acd8-78c3a47505cc service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:35:55.187276 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed4f0349-a335-41dc-acd8-78c3a47505cc service cinder] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:55.187625 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed4f0349-a335-41dc-acd8-78c3a47505cc service cinder] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:55.187914 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed4f0349-a335-41dc-acd8-78c3a47505cc service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:55.201131 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed4f0349-a335-41dc-acd8-78c3a47505cc service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:35:55.202603 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed4f0349-a335-41dc-acd8-78c3a47505cc service cinder] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:35:55.212412 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1295/2577] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:35:55 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:35:56.595129 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92f77f0c-4020-45e0-9e54-7380eaab7e67 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:56.595712 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92f77f0c-4020-45e0-9e54-7380eaab7e67 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:56.595712 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92f77f0c-4020-45e0-9e54-7380eaab7e67 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:56.615778 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-92f77f0c-4020-45e0-9e54-7380eaab7e67 None None] MFA Rules not processed for user `f757f2efd63f4526892784e78a402ccd`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:35:56.617272 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-92f77f0c-4020-45e0-9e54-7380eaab7e67 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:35:56.625489 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1283/2578] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:35:56 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:35:56.669212 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed4f0349-a335-41dc-acd8-78c3a47505cc service cinder] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:35:56.670961 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-ed4f0349-a335-41dc-acd8-78c3a47505cc service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:35:56.676774 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:35:56.744889 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed4f0349-a335-41dc-acd8-78c3a47505cc service cinder] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:35:56.749483 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddf28655-2b6b-4cfb-9bf6-d5775bdf96ad service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14176b54cf6640abb3b1c14e3824d4f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:35:56.750117 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf28655-2b6b-4cfb-9bf6-d5775bdf96ad service glance] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:56.750398 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf28655-2b6b-4cfb-9bf6-d5775bdf96ad service glance] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:56.750668 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf28655-2b6b-4cfb-9bf6-d5775bdf96ad service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:56.752823 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-ddf28655-2b6b-4cfb-9bf6-d5775bdf96ad service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:35:56.852042 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddf28655-2b6b-4cfb-9bf6-d5775bdf96ad service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:35:56.853458 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddf28655-2b6b-4cfb-9bf6-d5775bdf96ad service glance] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:35:56.875230 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1296/2579] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:35:56 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 207 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:35:57.357711 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92f77f0c-4020-45e0-9e54-7380eaab7e67 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:35:57.359142 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-92f77f0c-4020-45e0-9e54-7380eaab7e67 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:35:57.375707 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddf28655-2b6b-4cfb-9bf6-d5775bdf96ad service glance] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:35:57.377215 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-ddf28655-2b6b-4cfb-9bf6-d5775bdf96ad service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:35:57.417463 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:35:57.438470 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92f77f0c-4020-45e0-9e54-7380eaab7e67 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:35:57.443597 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddf28655-2b6b-4cfb-9bf6-d5775bdf96ad service glance] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:35:57.445285 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96a08d8e-03a6-49bb-ba27-2fc495370d7c service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2a3834fa6894dada3291e4781bbd553', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:35:57.445910 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96a08d8e-03a6-49bb-ba27-2fc495370d7c service placement] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:57.446190 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96a08d8e-03a6-49bb-ba27-2fc495370d7c service placement] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:57.446465 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96a08d8e-03a6-49bb-ba27-2fc495370d7c service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:57.447351 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7ea4cd2-f873-4c4b-a843-3f5934cb97a4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:35:57.447920 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7ea4cd2-f873-4c4b-a843-3f5934cb97a4 service neutron] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:35:57.448205 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7ea4cd2-f873-4c4b-a843-3f5934cb97a4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:35:57.448305 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-96a08d8e-03a6-49bb-ba27-2fc495370d7c service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:35:57.448470 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7ea4cd2-f873-4c4b-a843-3f5934cb97a4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:35:57.450327 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-c7ea4cd2-f873-4c4b-a843-3f5934cb97a4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:35:57.460934 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96a08d8e-03a6-49bb-ba27-2fc495370d7c service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:35:57.462522 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96a08d8e-03a6-49bb-ba27-2fc495370d7c service placement] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:35:57.464997 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7ea4cd2-f873-4c4b-a843-3f5934cb97a4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:35:57.466470 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7ea4cd2-f873-4c4b-a843-3f5934cb97a4 service neutron] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:35:57.473621 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1284/2580] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:35:57 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 116 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:35:57.478365 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1297/2581] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:35:57 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:36:00.219825 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96a08d8e-03a6-49bb-ba27-2fc495370d7c service placement] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:36:00.222143 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-96a08d8e-03a6-49bb-ba27-2fc495370d7c service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:36:00.253409 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96a08d8e-03a6-49bb-ba27-2fc495370d7c service placement] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:36:00.259866 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5ccf0e9-fee8-434e-a48c-76db1f7873f6 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '168b07df50c24593ba73dbde5b037aed', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:36:00.260504 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5ccf0e9-fee8-434e-a48c-76db1f7873f6 service nova] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:36:00.260781 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5ccf0e9-fee8-434e-a48c-76db1f7873f6 service nova] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:36:00.261091 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5ccf0e9-fee8-434e-a48c-76db1f7873f6 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:36:00.262967 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-e5ccf0e9-fee8-434e-a48c-76db1f7873f6 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:36:00.281912 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5ccf0e9-fee8-434e-a48c-76db1f7873f6 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:36:00.283734 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5ccf0e9-fee8-434e-a48c-76db1f7873f6 service nova] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:36:00.296370 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1285/2582] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:36:00 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 77 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:36:07.472990 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=91 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:36:07.812097 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7ea4cd2-f873-4c4b-a843-3f5934cb97a4 service neutron] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:36:07.813893 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-c7ea4cd2-f873-4c4b-a843-3f5934cb97a4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:36:07.829984 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7ea4cd2-f873-4c4b-a843-3f5934cb97a4 service neutron] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:36:07.836379 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-508c2d38-45ce-425e-bfdc-9b4cf4eaf2ba service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '168b07df50c24593ba73dbde5b037aed', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:36:07.837117 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-508c2d38-45ce-425e-bfdc-9b4cf4eaf2ba service nova] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:36:07.837524 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-508c2d38-45ce-425e-bfdc-9b4cf4eaf2ba service nova] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:36:07.837921 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-508c2d38-45ce-425e-bfdc-9b4cf4eaf2ba service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:36:07.840172 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-508c2d38-45ce-425e-bfdc-9b4cf4eaf2ba service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:36:07.858144 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-508c2d38-45ce-425e-bfdc-9b4cf4eaf2ba service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:36:07.861027 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-508c2d38-45ce-425e-bfdc-9b4cf4eaf2ba service nova] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:36:07.876124 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1298/2583] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:36:07 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 65 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:36:10.291654 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=15 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:36:14.598349 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bf817cd-2748-41df-91e4-54f48489d3c1 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:36:14.598959 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bf817cd-2748-41df-91e4-54f48489d3c1 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:36:14.598959 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bf817cd-2748-41df-91e4-54f48489d3c1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:36:14.672102 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-5bf817cd-2748-41df-91e4-54f48489d3c1 None None] MFA Rules not processed for user `eede08f020e04282bb994b855e371433`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:36:14.673782 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-5bf817cd-2748-41df-91e4-54f48489d3c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:36:14.744655 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1286/2584] 10.222.0.42 () {58 vars in 1009 bytes} [Sun Aug 28 01:36:14 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:36:14.803061 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-508c2d38-45ce-425e-bfdc-9b4cf4eaf2ba service nova] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:36:14.804696 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-508c2d38-45ce-425e-bfdc-9b4cf4eaf2ba service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:36:14.909849 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-508c2d38-45ce-425e-bfdc-9b4cf4eaf2ba service nova] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:36:14.915696 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0a1126b-89ed-4457-a5f1-be856a985d4e service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac4c5584582f4f55a7085af4cbbabf9b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:36:14.916302 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0a1126b-89ed-4457-a5f1-be856a985d4e service swift] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:36:14.916578 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0a1126b-89ed-4457-a5f1-be856a985d4e service swift] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:36:14.916846 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0a1126b-89ed-4457-a5f1-be856a985d4e service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:36:14.926157 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0a1126b-89ed-4457-a5f1-be856a985d4e service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:36:14.927698 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0a1126b-89ed-4457-a5f1-be856a985d4e service swift] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:36:14.937805 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1299/2585] 10.222.0.42 () {60 vars in 1384 bytes} [Sun Aug 28 01:36:14 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 135 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 28 01:36:15.487186 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4add0771-7035-47f7-ad10-7d41f379337b None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:36:15.487471 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4add0771-7035-47f7-ad10-7d41f379337b None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:36:15.487739 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4add0771-7035-47f7-ad10-7d41f379337b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:36:15.509739 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-4add0771-7035-47f7-ad10-7d41f379337b None None] MFA Rules not processed for user `eede08f020e04282bb994b855e371433`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:36:15.510968 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-4add0771-7035-47f7-ad10-7d41f379337b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:36:15.519024 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1287/2586] 10.222.0.42 () {58 vars in 967 bytes} [Sun Aug 28 01:36:15 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:36:15.534051 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0a1126b-89ed-4457-a5f1-be856a985d4e service swift] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:36:15.546733 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0a1126b-89ed-4457-a5f1-be856a985d4e service swift] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:36:15.551938 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab39d65e-6357-439a-ac55-c6c13774baf8 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac4c5584582f4f55a7085af4cbbabf9b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:36:15.552554 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab39d65e-6357-439a-ac55-c6c13774baf8 service swift] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:36:15.552833 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab39d65e-6357-439a-ac55-c6c13774baf8 service swift] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:36:15.553139 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab39d65e-6357-439a-ac55-c6c13774baf8 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:36:15.562949 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab39d65e-6357-439a-ac55-c6c13774baf8 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:36:15.564572 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab39d65e-6357-439a-ac55-c6c13774baf8 service swift] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:36:15.573671 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1300/2587] 10.222.0.42 () {60 vars in 1384 bytes} [Sun Aug 28 01:36:15 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 28 01:36:15.807789 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9909912f-14ec-417b-9775-2879f2f3f423 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:36:15.808240 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9909912f-14ec-417b-9775-2879f2f3f423 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:36:15.808333 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9909912f-14ec-417b-9775-2879f2f3f423 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:36:15.828087 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-9909912f-14ec-417b-9775-2879f2f3f423 None None] MFA Rules not processed for user `eede08f020e04282bb994b855e371433`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:36:15.829345 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-9909912f-14ec-417b-9775-2879f2f3f423 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:36:15.837215 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1288/2588] 10.222.0.42 () {58 vars in 1009 bytes} [Sun Aug 28 01:36:15 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:36:15.849654 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab39d65e-6357-439a-ac55-c6c13774baf8 service swift] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:36:15.861389 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab39d65e-6357-439a-ac55-c6c13774baf8 service swift] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:36:15.867463 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-955be503-335e-434c-9f55-2a5e4f1694cd service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac4c5584582f4f55a7085af4cbbabf9b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:36:15.868090 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-955be503-335e-434c-9f55-2a5e4f1694cd service swift] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:36:15.868372 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-955be503-335e-434c-9f55-2a5e4f1694cd service swift] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:36:15.868644 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-955be503-335e-434c-9f55-2a5e4f1694cd service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:36:15.878281 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-955be503-335e-434c-9f55-2a5e4f1694cd service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:36:15.879920 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-955be503-335e-434c-9f55-2a5e4f1694cd service swift] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:36:15.891951 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1301/2589] 10.222.0.42 () {60 vars in 1384 bytes} [Sun Aug 28 01:36:15 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 28 01:36:16.256508 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de8e24a-4c28-4357-b25b-ef5d2690b5bb None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:36:16.256793 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de8e24a-4c28-4357-b25b-ef5d2690b5bb None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:36:16.257097 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de8e24a-4c28-4357-b25b-ef5d2690b5bb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:36:16.277663 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-4de8e24a-4c28-4357-b25b-ef5d2690b5bb None None] MFA Rules not processed for user `eede08f020e04282bb994b855e371433`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:36:16.278922 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-4de8e24a-4c28-4357-b25b-ef5d2690b5bb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:36:16.286849 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1289/2590] 10.222.0.42 () {58 vars in 967 bytes} [Sun Aug 28 01:36:16 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:36:16.299377 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-955be503-335e-434c-9f55-2a5e4f1694cd service swift] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:36:16.311224 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-955be503-335e-434c-9f55-2a5e4f1694cd service swift] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:36:16.319948 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66893ff7-39c7-4dc3-b0db-4be9d778a28f service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac4c5584582f4f55a7085af4cbbabf9b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:36:16.320577 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66893ff7-39c7-4dc3-b0db-4be9d778a28f service swift] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:36:16.320992 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66893ff7-39c7-4dc3-b0db-4be9d778a28f service swift] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:36:16.321242 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66893ff7-39c7-4dc3-b0db-4be9d778a28f service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:36:16.331609 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66893ff7-39c7-4dc3-b0db-4be9d778a28f service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:36:16.333371 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66893ff7-39c7-4dc3-b0db-4be9d778a28f service swift] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:36:16.342776 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1302/2591] 10.222.0.42 () {60 vars in 1384 bytes} [Sun Aug 28 01:36:16 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 28 01:36:18.539772 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4de8e24a-4c28-4357-b25b-ef5d2690b5bb None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:36:18.609762 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4de8e24a-4c28-4357-b25b-ef5d2690b5bb None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:36:18.615271 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9fe791b-8704-479c-8b81-0e8f3b0c14cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:36:18.615903 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9fe791b-8704-479c-8b81-0e8f3b0c14cc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:36:18.616176 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9fe791b-8704-479c-8b81-0e8f3b0c14cc admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:36:18.616440 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9fe791b-8704-479c-8b81-0e8f3b0c14cc admin admin] PATH_INFO: `/v3/users/f757f2efd63f4526892784e78a402ccd` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:36:18.617780 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9fe791b-8704-479c-8b81-0e8f3b0c14cc admin admin] RBAC: Authorizing `identity:delete_user(user_id=f757f2efd63f4526892784e78a402ccd)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:36:18.619733 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9fe791b-8704-479c-8b81-0e8f3b0c14cc admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:36:18.671925 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=93,keystone:DELETE=7 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:36:18.673710 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-a9fe791b-8704-479c-8b81-0e8f3b0c14cc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f757f2efd63f4526892784e78a402ccd'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:36:18.691276 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-a9fe791b-8704-479c-8b81-0e8f3b0c14cc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f757f2efd63f4526892784e78a402ccd'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:36:18.701070 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-a9fe791b-8704-479c-8b81-0e8f3b0c14cc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f757f2efd63f4526892784e78a402ccd'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:36:18.708947 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-a9fe791b-8704-479c-8b81-0e8f3b0c14cc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f757f2efd63f4526892784e78a402ccd'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:36:18.709892 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-a9fe791b-8704-479c-8b81-0e8f3b0c14cc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f757f2efd63f4526892784e78a402ccd'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:36:18.716419 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1290/2592] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 01:36:18 2022] DELETE /identity/v3/users/f757f2efd63f4526892784e78a402ccd => generated 0 bytes in 177 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:36:18.721406 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aafc389-44fc-41e1-8840-d7ad4e6cb95a None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:36:18.721965 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aafc389-44fc-41e1-8840-d7ad4e6cb95a None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:36:18.721965 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aafc389-44fc-41e1-8840-d7ad4e6cb95a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:36:18.741803 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-4aafc389-44fc-41e1-8840-d7ad4e6cb95a None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:36:18.743014 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-4aafc389-44fc-41e1-8840-d7ad4e6cb95a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:36:18.802054 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1303/2593] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:36:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:36:18.813870 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9fe791b-8704-479c-8b81-0e8f3b0c14cc admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:36:18.815663 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-a9fe791b-8704-479c-8b81-0e8f3b0c14cc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:36:18.886285 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9fe791b-8704-479c-8b81-0e8f3b0c14cc admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:36:18.890278 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df2a8cd3-b1ce-4d88-bad2-0adb0b10afe8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:36:18.890887 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df2a8cd3-b1ce-4d88-bad2-0adb0b10afe8 service neutron] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:36:18.891161 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df2a8cd3-b1ce-4d88-bad2-0adb0b10afe8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:36:18.891413 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df2a8cd3-b1ce-4d88-bad2-0adb0b10afe8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:36:18.901129 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df2a8cd3-b1ce-4d88-bad2-0adb0b10afe8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:36:18.902582 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df2a8cd3-b1ce-4d88-bad2-0adb0b10afe8 service neutron] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:36:18.912558 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1291/2594] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:36:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:36:19.094782 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aafc389-44fc-41e1-8840-d7ad4e6cb95a None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:36:19.096321 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-4aafc389-44fc-41e1-8840-d7ad4e6cb95a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:36:19.108644 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aafc389-44fc-41e1-8840-d7ad4e6cb95a None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:36:19.115164 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ebd1332-c47c-418b-8e9e-524fc11e8fe2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:36:19.116132 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ebd1332-c47c-418b-8e9e-524fc11e8fe2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:36:19.116591 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ebd1332-c47c-418b-8e9e-524fc11e8fe2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:36:19.117092 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ebd1332-c47c-418b-8e9e-524fc11e8fe2 admin admin] PATH_INFO: `/v3/projects/802f856adff04f85b6731d6a97331da6` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:36:19.118786 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ebd1332-c47c-418b-8e9e-524fc11e8fe2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=802f856adff04f85b6731d6a97331da6)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:36:19.120755 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ebd1332-c47c-418b-8e9e-524fc11e8fe2 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:36:19.134464 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:36:19.176391 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-6ebd1332-c47c-418b-8e9e-524fc11e8fe2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '802f856adff04f85b6731d6a97331da6', 'request_id': 'req-6ebd1332-c47c-418b-8e9e-524fc11e8fe2'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:36:19.182943 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-6ebd1332-c47c-418b-8e9e-524fc11e8fe2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '802f856adff04f85b6731d6a97331da6', 'request_id': 'req-6ebd1332-c47c-418b-8e9e-524fc11e8fe2'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:36:19.189730 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-6ebd1332-c47c-418b-8e9e-524fc11e8fe2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '802f856adff04f85b6731d6a97331da6', 'request_id': 'req-6ebd1332-c47c-418b-8e9e-524fc11e8fe2'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:36:19.206363 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-6ebd1332-c47c-418b-8e9e-524fc11e8fe2 admin admin] The token cache is being invalidate because project 802f856adff04f85b6731d6a97331da6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81508) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 01:36:19.206656 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-6ebd1332-c47c-418b-8e9e-524fc11e8fe2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:36:19.207320 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-6ebd1332-c47c-418b-8e9e-524fc11e8fe2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:36:19.208892 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1304/2595] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 01:36:19 2022] DELETE /identity/v3/projects/802f856adff04f85b6731d6a97331da6 => generated 0 bytes in 115 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:36:19.214134 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c649bf6a-bf14-42f6-bc7d-ab7eb15097ca None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:36:19.214424 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c649bf6a-bf14-42f6-bc7d-ab7eb15097ca None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:36:19.214697 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c649bf6a-bf14-42f6-bc7d-ab7eb15097ca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:36:19.237575 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-c649bf6a-bf14-42f6-bc7d-ab7eb15097ca None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:36:19.239423 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-c649bf6a-bf14-42f6-bc7d-ab7eb15097ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:36:19.315244 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1292/2596] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:36:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 28 01:36:19.325247 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c56e4622-aa49-4849-9263-dd1d4743a9c7 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:36:19.325557 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c56e4622-aa49-4849-9263-dd1d4743a9c7 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:36:19.325815 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c56e4622-aa49-4849-9263-dd1d4743a9c7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:36:19.346826 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-c56e4622-aa49-4849-9263-dd1d4743a9c7 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:36:19.347988 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-c56e4622-aa49-4849-9263-dd1d4743a9c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:36:19.356719 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1305/2597] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:36:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:36:19.361218 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c649bf6a-bf14-42f6-bc7d-ab7eb15097ca None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:36:19.371431 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c649bf6a-bf14-42f6-bc7d-ab7eb15097ca None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:36:19.376375 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d760c1d7-c3e9-4883-aef0-9aa6e8e776dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:36:19.376991 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d760c1d7-c3e9-4883-aef0-9aa6e8e776dd admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:36:19.377301 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d760c1d7-c3e9-4883-aef0-9aa6e8e776dd admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:36:19.377560 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d760c1d7-c3e9-4883-aef0-9aa6e8e776dd admin admin] PATH_INFO: `/v3/domains` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:36:19.378212 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d760c1d7-c3e9-4883-aef0-9aa6e8e776dd admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 01:36:19.378512 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d760c1d7-c3e9-4883-aef0-9aa6e8e776dd admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:36:19.380125 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d760c1d7-c3e9-4883-aef0-9aa6e8e776dd admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:36:19.392484 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1293/2598] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Aug 28 01:36:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 28 01:36:19.397260 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c56e4622-aa49-4849-9263-dd1d4743a9c7 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:36:19.410810 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c56e4622-aa49-4849-9263-dd1d4743a9c7 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:36:19.418354 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba8580e8-9395-440a-bee3-e33698b67c3c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:36:19.419261 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba8580e8-9395-440a-bee3-e33698b67c3c admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:36:19.419627 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba8580e8-9395-440a-bee3-e33698b67c3c admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:36:19.419987 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba8580e8-9395-440a-bee3-e33698b67c3c admin admin] PATH_INFO: `/v3/projects` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:36:19.420891 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba8580e8-9395-440a-bee3-e33698b67c3c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:36:19.423923 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba8580e8-9395-440a-bee3-e33698b67c3c admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:36:19.463031 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1306/2599] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 01:36:19 2022] POST /identity/v3/projects => generated 397 bytes in 66 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 01:36:19.469572 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d760c1d7-c3e9-4883-aef0-9aa6e8e776dd admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:36:19.482677 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d760c1d7-c3e9-4883-aef0-9aa6e8e776dd admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:36:19.525137 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=96,keystone:DELETE=3,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:36:19.543489 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52b2ee5b-30a9-4547-a4e5-56443463e50f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:36:19.544100 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52b2ee5b-30a9-4547-a4e5-56443463e50f admin admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:36:19.544399 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52b2ee5b-30a9-4547-a4e5-56443463e50f admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:36:19.544666 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52b2ee5b-30a9-4547-a4e5-56443463e50f admin admin] PATH_INFO: `/v3/users` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:36:19.545400 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52b2ee5b-30a9-4547-a4e5-56443463e50f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:36:19.547405 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52b2ee5b-30a9-4547-a4e5-56443463e50f admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:36:19.573514 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1294/2600] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 01:36:19 2022] POST /identity/v3/users => generated 299 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 01:36:19.577946 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba8580e8-9395-440a-bee3-e33698b67c3c admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:36:19.589000 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba8580e8-9395-440a-bee3-e33698b67c3c admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:36:19.594227 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89382d82-b41d-4983-8894-026a57daa01a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:36:19.594815 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89382d82-b41d-4983-8894-026a57daa01a admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:36:19.595091 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89382d82-b41d-4983-8894-026a57daa01a admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:36:19.595359 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89382d82-b41d-4983-8894-026a57daa01a admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:36:19.595990 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89382d82-b41d-4983-8894-026a57daa01a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:36:19.597686 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89382d82-b41d-4983-8894-026a57daa01a admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:36:19.607520 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1307/2601] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 01:36:19 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 01:36:19.614076 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52b2ee5b-30a9-4547-a4e5-56443463e50f admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:36:19.626859 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52b2ee5b-30a9-4547-a4e5-56443463e50f admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:36:19.633683 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26e0b931-4b0b-4dea-9257-d5850fc1fc1a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:36:19.634321 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e0b931-4b0b-4dea-9257-d5850fc1fc1a admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:36:19.634598 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e0b931-4b0b-4dea-9257-d5850fc1fc1a admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:36:19.634864 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e0b931-4b0b-4dea-9257-d5850fc1fc1a admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:36:19.635522 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26e0b931-4b0b-4dea-9257-d5850fc1fc1a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:36:19.637197 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26e0b931-4b0b-4dea-9257-d5850fc1fc1a admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:36:19.649988 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1295/2602] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 01:36:19 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 01:36:19.655271 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89382d82-b41d-4983-8894-026a57daa01a admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:36:19.668670 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89382d82-b41d-4983-8894-026a57daa01a admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:36:19.674069 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acc52845-0776-4aed-9d38-60fcfda020cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:36:19.674756 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acc52845-0776-4aed-9d38-60fcfda020cf admin admin] REQUEST_METHOD: `PUT` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:36:19.675063 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acc52845-0776-4aed-9d38-60fcfda020cf admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:36:19.675334 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acc52845-0776-4aed-9d38-60fcfda020cf admin admin] PATH_INFO: `/v3/projects/3a87506e921f46739b918aaa06b84107/users/4780d9d6b88d4999b0536db8da5b24f7/roles/0487ef7e943e4701915cb8ce0f2899d7` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:36:19.693738 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acc52845-0776-4aed-9d38-60fcfda020cf admin admin] RBAC: Authorizing `identity:create_grant(project_id=3a87506e921f46739b918aaa06b84107, user_id=4780d9d6b88d4999b0536db8da5b24f7, role_id=0487ef7e943e4701915cb8ce0f2899d7)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:36:19.696884 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acc52845-0776-4aed-9d38-60fcfda020cf admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:36:19.714941 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1308/2603] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 01:36:19 2022] PUT /identity/v3/projects/3a87506e921f46739b918aaa06b84107/users/4780d9d6b88d4999b0536db8da5b24f7/roles/0487ef7e943e4701915cb8ce0f2899d7 => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:36:19.726592 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86032a0a-bfa7-47d5-bc96-b0ad4bb27df5 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:36:19.726864 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86032a0a-bfa7-47d5-bc96-b0ad4bb27df5 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:36:19.727134 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86032a0a-bfa7-47d5-bc96-b0ad4bb27df5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:36:19.766251 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-86032a0a-bfa7-47d5-bc96-b0ad4bb27df5 None None] MFA Rules not processed for user `4780d9d6b88d4999b0536db8da5b24f7`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:36:19.766251 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-86032a0a-bfa7-47d5-bc96-b0ad4bb27df5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:36:19.827583 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1296/2604] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:36:19 2022] POST /identity/v3/auth/tokens => generated 3274 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:36:19.839824 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acc52845-0776-4aed-9d38-60fcfda020cf admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:36:19.841687 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-acc52845-0776-4aed-9d38-60fcfda020cf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:36:19.906083 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acc52845-0776-4aed-9d38-60fcfda020cf admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:36:19.910301 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f146833a-bc07-4d10-94f8-af2edf72fc68 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:36:19.910912 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f146833a-bc07-4d10-94f8-af2edf72fc68 service cinder] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:36:19.911187 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f146833a-bc07-4d10-94f8-af2edf72fc68 service cinder] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:36:19.911454 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f146833a-bc07-4d10-94f8-af2edf72fc68 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:36:19.920276 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f146833a-bc07-4d10-94f8-af2edf72fc68 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:36:19.921751 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f146833a-bc07-4d10-94f8-af2edf72fc68 service cinder] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:36:19.933053 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1309/2605] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:36:19 2022] GET /identity/v3/auth/tokens => generated 3274 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:36:28.116715 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86032a0a-bfa7-47d5-bc96-b0ad4bb27df5 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:36:28.118250 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-86032a0a-bfa7-47d5-bc96-b0ad4bb27df5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:36:28.177207 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86032a0a-bfa7-47d5-bc96-b0ad4bb27df5 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:36:28.181741 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1638c8ba-cd17-4508-8aee-ddb0eb8d1961 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:36:28.182355 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1638c8ba-cd17-4508-8aee-ddb0eb8d1961 service neutron] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:36:28.182624 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1638c8ba-cd17-4508-8aee-ddb0eb8d1961 service neutron] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:36:28.182897 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1638c8ba-cd17-4508-8aee-ddb0eb8d1961 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:36:28.185148 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-1638c8ba-cd17-4508-8aee-ddb0eb8d1961 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:36:28.195855 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1638c8ba-cd17-4508-8aee-ddb0eb8d1961 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:36:28.197323 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1638c8ba-cd17-4508-8aee-ddb0eb8d1961 service neutron] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:36:28.205597 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1297/2606] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:36:28 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:36:29.140308 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f146833a-bc07-4d10-94f8-af2edf72fc68 service cinder] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:36:29.187143 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:36:29.211507 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f146833a-bc07-4d10-94f8-af2edf72fc68 service cinder] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:36:29.216472 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f865d17f-075b-4675-b5ce-1047cba0ec2e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:36:29.217218 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f865d17f-075b-4675-b5ce-1047cba0ec2e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:36:29.217517 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f865d17f-075b-4675-b5ce-1047cba0ec2e admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:36:29.217794 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f865d17f-075b-4675-b5ce-1047cba0ec2e admin admin] PATH_INFO: `/v3/users/4780d9d6b88d4999b0536db8da5b24f7` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:36:29.218972 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f865d17f-075b-4675-b5ce-1047cba0ec2e admin admin] RBAC: Authorizing `identity:delete_user(user_id=4780d9d6b88d4999b0536db8da5b24f7)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:36:29.220917 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f865d17f-075b-4675-b5ce-1047cba0ec2e admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:36:29.273213 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-f865d17f-075b-4675-b5ce-1047cba0ec2e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4780d9d6b88d4999b0536db8da5b24f7'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:36:29.273854 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-f865d17f-075b-4675-b5ce-1047cba0ec2e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4780d9d6b88d4999b0536db8da5b24f7'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:36:29.282482 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-f865d17f-075b-4675-b5ce-1047cba0ec2e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4780d9d6b88d4999b0536db8da5b24f7'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:36:29.292452 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-f865d17f-075b-4675-b5ce-1047cba0ec2e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4780d9d6b88d4999b0536db8da5b24f7'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:36:29.293160 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-f865d17f-075b-4675-b5ce-1047cba0ec2e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4780d9d6b88d4999b0536db8da5b24f7'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:36:29.310756 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1310/2607] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 01:36:29 2022] DELETE /identity/v3/users/4780d9d6b88d4999b0536db8da5b24f7 => generated 0 bytes in 171 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:36:29.315408 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd59ac9d-2c60-430e-8724-2de49669ddca None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:36:29.315884 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd59ac9d-2c60-430e-8724-2de49669ddca None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:36:29.315982 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd59ac9d-2c60-430e-8724-2de49669ddca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:36:29.334719 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-cd59ac9d-2c60-430e-8724-2de49669ddca None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:36:29.335869 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-cd59ac9d-2c60-430e-8724-2de49669ddca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:36:29.393044 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:36:29.410119 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1298/2608] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:36:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:36:29.420028 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f865d17f-075b-4675-b5ce-1047cba0ec2e admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:36:29.421589 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-f865d17f-075b-4675-b5ce-1047cba0ec2e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:36:29.484232 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f865d17f-075b-4675-b5ce-1047cba0ec2e admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:36:29.488859 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8f291a4-59cc-481b-857c-f6bbdaec938f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:36:29.489479 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8f291a4-59cc-481b-857c-f6bbdaec938f service neutron] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:36:29.489754 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8f291a4-59cc-481b-857c-f6bbdaec938f service neutron] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:36:29.490022 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8f291a4-59cc-481b-857c-f6bbdaec938f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:36:29.499703 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8f291a4-59cc-481b-857c-f6bbdaec938f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:36:29.501184 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8f291a4-59cc-481b-857c-f6bbdaec938f service neutron] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:36:29.511077 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1311/2609] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:36:29 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:36:29.814314 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd59ac9d-2c60-430e-8724-2de49669ddca None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:36:29.815741 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-cd59ac9d-2c60-430e-8724-2de49669ddca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:36:29.829084 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd59ac9d-2c60-430e-8724-2de49669ddca None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:36:29.838667 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb7833ce-a553-41e3-b662-218670fe28d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:36:29.839321 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb7833ce-a553-41e3-b662-218670fe28d8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:36:29.839602 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb7833ce-a553-41e3-b662-218670fe28d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:36:29.840008 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb7833ce-a553-41e3-b662-218670fe28d8 admin admin] PATH_INFO: `/v3/projects/3a87506e921f46739b918aaa06b84107` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:36:29.841207 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb7833ce-a553-41e3-b662-218670fe28d8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3a87506e921f46739b918aaa06b84107)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:36:29.843175 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb7833ce-a553-41e3-b662-218670fe28d8 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:36:29.904929 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-eb7833ce-a553-41e3-b662-218670fe28d8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3a87506e921f46739b918aaa06b84107', 'request_id': 'req-eb7833ce-a553-41e3-b662-218670fe28d8'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:36:29.911896 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-eb7833ce-a553-41e3-b662-218670fe28d8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3a87506e921f46739b918aaa06b84107', 'request_id': 'req-eb7833ce-a553-41e3-b662-218670fe28d8'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:36:29.918620 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-eb7833ce-a553-41e3-b662-218670fe28d8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3a87506e921f46739b918aaa06b84107', 'request_id': 'req-eb7833ce-a553-41e3-b662-218670fe28d8'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:36:29.933622 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-eb7833ce-a553-41e3-b662-218670fe28d8 admin admin] The token cache is being invalidate because project 3a87506e921f46739b918aaa06b84107 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81507) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 01:36:29.933900 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-eb7833ce-a553-41e3-b662-218670fe28d8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:36:29.934556 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-eb7833ce-a553-41e3-b662-218670fe28d8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:36:29.936154 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1299/2610] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 01:36:29 2022] DELETE /identity/v3/projects/3a87506e921f46739b918aaa06b84107 => generated 0 bytes in 122 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:36:29.942608 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1721418-a106-48e1-9bc7-306d65811fd1 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:36:29.943123 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1721418-a106-48e1-9bc7-306d65811fd1 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:36:29.943460 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1721418-a106-48e1-9bc7-306d65811fd1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:36:29.966932 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-d1721418-a106-48e1-9bc7-306d65811fd1 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:36:29.968223 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-d1721418-a106-48e1-9bc7-306d65811fd1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:36:30.036692 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1312/2611] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:36:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:36:30.044556 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcf65461-079a-4573-b447-be5445569fe1 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:36:30.044837 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcf65461-079a-4573-b447-be5445569fe1 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:36:30.045146 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcf65461-079a-4573-b447-be5445569fe1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:36:30.066170 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-dcf65461-079a-4573-b447-be5445569fe1 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:36:30.067380 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-dcf65461-079a-4573-b447-be5445569fe1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:36:30.076470 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1300/2612] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:36:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:36:30.080879 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1721418-a106-48e1-9bc7-306d65811fd1 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:36:30.085949 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:36:30.093013 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1721418-a106-48e1-9bc7-306d65811fd1 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:36:30.099951 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21cf8f69-2a54-4a7c-a02c-f76fbb001248 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:36:30.100568 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21cf8f69-2a54-4a7c-a02c-f76fbb001248 admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:36:30.101087 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21cf8f69-2a54-4a7c-a02c-f76fbb001248 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:36:30.102176 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21cf8f69-2a54-4a7c-a02c-f76fbb001248 admin admin] PATH_INFO: `/v3/domains` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:36:30.103041 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21cf8f69-2a54-4a7c-a02c-f76fbb001248 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 01:36:30.103366 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21cf8f69-2a54-4a7c-a02c-f76fbb001248 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:36:30.105157 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21cf8f69-2a54-4a7c-a02c-f76fbb001248 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:36:30.121257 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1313/2613] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Aug 28 01:36:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 01:36:30.125888 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcf65461-079a-4573-b447-be5445569fe1 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:36:30.135627 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcf65461-079a-4573-b447-be5445569fe1 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:36:30.141255 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03c9c444-6b62-4460-80d7-2328964d80c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:36:30.142064 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03c9c444-6b62-4460-80d7-2328964d80c5 admin admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:36:30.142514 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03c9c444-6b62-4460-80d7-2328964d80c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:36:30.142641 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03c9c444-6b62-4460-80d7-2328964d80c5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:36:30.143242 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03c9c444-6b62-4460-80d7-2328964d80c5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:36:30.145200 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03c9c444-6b62-4460-80d7-2328964d80c5 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:36:30.178565 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1301/2614] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 01:36:30 2022] POST /identity/v3/projects => generated 385 bytes in 53 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 01:36:30.183351 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21cf8f69-2a54-4a7c-a02c-f76fbb001248 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:36:30.194565 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21cf8f69-2a54-4a7c-a02c-f76fbb001248 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:36:30.254335 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9f72419-c755-486e-a0eb-938950404107 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:36:30.254963 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9f72419-c755-486e-a0eb-938950404107 admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:36:30.255236 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9f72419-c755-486e-a0eb-938950404107 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:36:30.255500 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9f72419-c755-486e-a0eb-938950404107 admin admin] PATH_INFO: `/v3/users` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:36:30.256127 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9f72419-c755-486e-a0eb-938950404107 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:36:30.258104 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9f72419-c755-486e-a0eb-938950404107 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:36:30.282698 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1314/2615] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 01:36:30 2022] POST /identity/v3/users => generated 293 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 01:36:30.287246 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03c9c444-6b62-4460-80d7-2328964d80c5 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:36:30.297867 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03c9c444-6b62-4460-80d7-2328964d80c5 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:36:30.302922 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70ab4a91-1777-43df-9b14-fe3a9b6c7218 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:36:30.303525 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70ab4a91-1777-43df-9b14-fe3a9b6c7218 admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:36:30.303823 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70ab4a91-1777-43df-9b14-fe3a9b6c7218 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:36:30.304063 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70ab4a91-1777-43df-9b14-fe3a9b6c7218 admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:36:30.304701 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70ab4a91-1777-43df-9b14-fe3a9b6c7218 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:36:30.306703 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70ab4a91-1777-43df-9b14-fe3a9b6c7218 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:36:30.317234 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1302/2616] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 01:36:30 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 28 01:36:30.321416 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9f72419-c755-486e-a0eb-938950404107 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:36:30.331829 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9f72419-c755-486e-a0eb-938950404107 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:36:30.336722 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cefa6ee-59a5-446b-9625-98c9c397d682 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:36:30.337349 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cefa6ee-59a5-446b-9625-98c9c397d682 admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:36:30.337626 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cefa6ee-59a5-446b-9625-98c9c397d682 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:36:30.337895 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cefa6ee-59a5-446b-9625-98c9c397d682 admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:36:30.338530 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cefa6ee-59a5-446b-9625-98c9c397d682 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:36:30.340160 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cefa6ee-59a5-446b-9625-98c9c397d682 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:36:30.351491 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1315/2617] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 01:36:30 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 01:36:30.355818 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70ab4a91-1777-43df-9b14-fe3a9b6c7218 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:36:30.366522 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70ab4a91-1777-43df-9b14-fe3a9b6c7218 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:36:30.371813 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f0d4ecd-abad-428d-b690-82de33080c2e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:36:30.372461 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f0d4ecd-abad-428d-b690-82de33080c2e admin admin] REQUEST_METHOD: `PUT` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:36:30.372739 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f0d4ecd-abad-428d-b690-82de33080c2e admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:36:30.373001 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f0d4ecd-abad-428d-b690-82de33080c2e admin admin] PATH_INFO: `/v3/projects/8bd4256c51f64ef9aa0f21ea2fcb55e7/users/c5bc73cffe1e4fa095051e109b5f0d0c/roles/0487ef7e943e4701915cb8ce0f2899d7` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:36:30.388644 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f0d4ecd-abad-428d-b690-82de33080c2e admin admin] RBAC: Authorizing `identity:create_grant(project_id=8bd4256c51f64ef9aa0f21ea2fcb55e7, user_id=c5bc73cffe1e4fa095051e109b5f0d0c, role_id=0487ef7e943e4701915cb8ce0f2899d7)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:36:30.391339 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f0d4ecd-abad-428d-b690-82de33080c2e admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:36:30.407497 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1303/2618] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 01:36:30 2022] PUT /identity/v3/projects/8bd4256c51f64ef9aa0f21ea2fcb55e7/users/c5bc73cffe1e4fa095051e109b5f0d0c/roles/0487ef7e943e4701915cb8ce0f2899d7 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:36:30.412571 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abfaf15b-d92e-474b-aee5-5b19303718fc None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:36:30.412857 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abfaf15b-d92e-474b-aee5-5b19303718fc None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:36:30.413178 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abfaf15b-d92e-474b-aee5-5b19303718fc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:36:30.432312 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-abfaf15b-d92e-474b-aee5-5b19303718fc None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:36:30.433591 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-abfaf15b-d92e-474b-aee5-5b19303718fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:36:30.502055 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1316/2619] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:36:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:36:30.515964 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f0d4ecd-abad-428d-b690-82de33080c2e admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:36:30.518147 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-8f0d4ecd-abad-428d-b690-82de33080c2e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:36:30.586962 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f0d4ecd-abad-428d-b690-82de33080c2e admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:36:30.591717 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c2b6f4d-2898-47d8-9fbe-b077081b814d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:36:30.592424 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c2b6f4d-2898-47d8-9fbe-b077081b814d service neutron] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:36:30.592717 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c2b6f4d-2898-47d8-9fbe-b077081b814d service neutron] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:36:30.593147 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c2b6f4d-2898-47d8-9fbe-b077081b814d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:36:30.604522 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c2b6f4d-2898-47d8-9fbe-b077081b814d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:36:30.606041 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c2b6f4d-2898-47d8-9fbe-b077081b814d service neutron] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:36:30.618924 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1304/2620] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:36:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:36:34.702537 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba46fb1d-0ea0-43bf-99a3-447efea84ca2 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:36:34.703057 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba46fb1d-0ea0-43bf-99a3-447efea84ca2 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:36:34.703057 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba46fb1d-0ea0-43bf-99a3-447efea84ca2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:36:34.737717 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-ba46fb1d-0ea0-43bf-99a3-447efea84ca2 None None] MFA Rules not processed for user `c5bc73cffe1e4fa095051e109b5f0d0c`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:36:34.738985 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-ba46fb1d-0ea0-43bf-99a3-447efea84ca2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:36:34.806765 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:36:34.812445 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1317/2621] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:36:34 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:36:34.817446 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c2b6f4d-2898-47d8-9fbe-b077081b814d service neutron] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:36:34.827327 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c2b6f4d-2898-47d8-9fbe-b077081b814d service neutron] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:36:34.832219 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69e69bd0-5861-497f-9e27-fabaf3fd7698 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:36:34.833012 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e69bd0-5861-497f-9e27-fabaf3fd7698 admin admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:36:34.833366 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e69bd0-5861-497f-9e27-fabaf3fd7698 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:36:34.833565 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e69bd0-5861-497f-9e27-fabaf3fd7698 admin admin] PATH_INFO: `/v3/projects` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:36:34.834216 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69e69bd0-5861-497f-9e27-fabaf3fd7698 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:36:34.836142 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69e69bd0-5861-497f-9e27-fabaf3fd7698 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:36:34.853268 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:36:34.868286 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1305/2622] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 01:36:34 2022] POST /identity/v3/projects => generated 385 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 01:36:34.873791 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba46fb1d-0ea0-43bf-99a3-447efea84ca2 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:36:34.888560 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba46fb1d-0ea0-43bf-99a3-447efea84ca2 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:36:34.952748 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a0c3857-c8c7-41c4-9441-fdc828f3a85e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:36:34.953404 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a0c3857-c8c7-41c4-9441-fdc828f3a85e admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:36:34.953683 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a0c3857-c8c7-41c4-9441-fdc828f3a85e admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:36:34.953954 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a0c3857-c8c7-41c4-9441-fdc828f3a85e admin admin] PATH_INFO: `/v3/users` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:36:34.954591 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a0c3857-c8c7-41c4-9441-fdc828f3a85e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:36:34.956519 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a0c3857-c8c7-41c4-9441-fdc828f3a85e admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:36:34.982492 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1318/2623] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 01:36:34 2022] POST /identity/v3/users => generated 299 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 01:36:34.987747 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69e69bd0-5861-497f-9e27-fabaf3fd7698 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:36:34.999223 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69e69bd0-5861-497f-9e27-fabaf3fd7698 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:36:35.004586 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90201ffc-5adc-4a92-a787-5594ad0c3427 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:36:35.005259 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90201ffc-5adc-4a92-a787-5594ad0c3427 admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:36:35.005680 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90201ffc-5adc-4a92-a787-5594ad0c3427 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:36:35.005962 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90201ffc-5adc-4a92-a787-5594ad0c3427 admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:36:35.006964 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90201ffc-5adc-4a92-a787-5594ad0c3427 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:36:35.008665 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90201ffc-5adc-4a92-a787-5594ad0c3427 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:36:35.023944 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1306/2624] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 01:36:34 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 01:36:35.028165 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a0c3857-c8c7-41c4-9441-fdc828f3a85e admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:36:35.056265 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a0c3857-c8c7-41c4-9441-fdc828f3a85e admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:36:35.066394 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94241045-d004-44b7-a137-f0af3822b636 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:36:35.067176 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94241045-d004-44b7-a137-f0af3822b636 admin admin] REQUEST_METHOD: `PUT` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:36:35.067511 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94241045-d004-44b7-a137-f0af3822b636 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:36:35.067849 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94241045-d004-44b7-a137-f0af3822b636 admin admin] PATH_INFO: `/v3/projects/52b4d06d235646a8a3b7074b559dba9c/users/300220ede3bc4d67a443eeab0d6511b2/roles/f08d6043261747e3b4763c744eef7dfb` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:36:35.084914 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94241045-d004-44b7-a137-f0af3822b636 admin admin] RBAC: Authorizing `identity:create_grant(project_id=52b4d06d235646a8a3b7074b559dba9c, user_id=300220ede3bc4d67a443eeab0d6511b2, role_id=f08d6043261747e3b4763c744eef7dfb)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:36:35.089417 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94241045-d004-44b7-a137-f0af3822b636 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:36:35.105604 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1319/2625] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 01:36:35 2022] PUT /identity/v3/projects/52b4d06d235646a8a3b7074b559dba9c/users/300220ede3bc4d67a443eeab0d6511b2/roles/f08d6043261747e3b4763c744eef7dfb => generated 0 bytes in 78 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:36:35.113997 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90201ffc-5adc-4a92-a787-5594ad0c3427 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:36:35.120953 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90201ffc-5adc-4a92-a787-5594ad0c3427 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:36:35.188089 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8916b86c-6e63-48c2-8836-3f1b2d36e67c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:36:35.188742 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8916b86c-6e63-48c2-8836-3f1b2d36e67c admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:36:35.189072 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8916b86c-6e63-48c2-8836-3f1b2d36e67c admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:36:35.189345 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8916b86c-6e63-48c2-8836-3f1b2d36e67c admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:36:35.189999 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8916b86c-6e63-48c2-8836-3f1b2d36e67c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:36:35.191638 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8916b86c-6e63-48c2-8836-3f1b2d36e67c admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:36:35.203437 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1307/2626] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 01:36:35 2022] GET /identity/v3/roles => generated 1420 bytes in 94 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 01:36:35.207924 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94241045-d004-44b7-a137-f0af3822b636 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:36:35.219098 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94241045-d004-44b7-a137-f0af3822b636 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:36:35.223847 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7122d83b-552e-4da8-8e7e-bf89c6eb3c00 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:36:35.224530 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7122d83b-552e-4da8-8e7e-bf89c6eb3c00 admin admin] REQUEST_METHOD: `PUT` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:36:35.224813 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7122d83b-552e-4da8-8e7e-bf89c6eb3c00 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:36:35.225116 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7122d83b-552e-4da8-8e7e-bf89c6eb3c00 admin admin] PATH_INFO: `/v3/projects/52b4d06d235646a8a3b7074b559dba9c/users/300220ede3bc4d67a443eeab0d6511b2/roles/f08d6043261747e3b4763c744eef7dfb` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:36:35.227045 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7122d83b-552e-4da8-8e7e-bf89c6eb3c00 admin admin] RBAC: Authorizing `identity:create_grant(project_id=52b4d06d235646a8a3b7074b559dba9c, user_id=300220ede3bc4d67a443eeab0d6511b2, role_id=f08d6043261747e3b4763c744eef7dfb)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:36:35.229738 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7122d83b-552e-4da8-8e7e-bf89c6eb3c00 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:36:35.244473 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1320/2627] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 01:36:35 2022] PUT /identity/v3/projects/52b4d06d235646a8a3b7074b559dba9c/users/300220ede3bc4d67a443eeab0d6511b2/roles/f08d6043261747e3b4763c744eef7dfb => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:36:39.624404 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a82f34e-9ffa-4e8f-8c34-a099232836b4 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:36:39.624932 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a82f34e-9ffa-4e8f-8c34-a099232836b4 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:36:39.624932 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a82f34e-9ffa-4e8f-8c34-a099232836b4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:36:39.660339 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-1a82f34e-9ffa-4e8f-8c34-a099232836b4 None None] MFA Rules not processed for user `300220ede3bc4d67a443eeab0d6511b2`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:36:39.661757 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-1a82f34e-9ffa-4e8f-8c34-a099232836b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:36:39.733785 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1308/2628] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:36:39 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:36:39.746155 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7122d83b-552e-4da8-8e7e-bf89c6eb3c00 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:36:39.748107 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-7122d83b-552e-4da8-8e7e-bf89c6eb3c00 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:36:39.856153 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7122d83b-552e-4da8-8e7e-bf89c6eb3c00 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:36:39.860300 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e90bbf9-f438-42dc-b92e-a6a9fa32775d service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14176b54cf6640abb3b1c14e3824d4f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:36:39.860935 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e90bbf9-f438-42dc-b92e-a6a9fa32775d service glance] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:36:39.861247 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e90bbf9-f438-42dc-b92e-a6a9fa32775d service glance] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:36:39.861520 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e90bbf9-f438-42dc-b92e-a6a9fa32775d service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:36:39.928905 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e90bbf9-f438-42dc-b92e-a6a9fa32775d service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:36:39.931047 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e90bbf9-f438-42dc-b92e-a6a9fa32775d service glance] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:36:39.941952 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1321/2629] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:36:39 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 196 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:36:40.087825 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-475b6572-3c1c-4e11-aa96-44bfaba96153 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:36:40.088106 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-475b6572-3c1c-4e11-aa96-44bfaba96153 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:36:40.088388 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-475b6572-3c1c-4e11-aa96-44bfaba96153 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:36:40.113318 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-475b6572-3c1c-4e11-aa96-44bfaba96153 None None] MFA Rules not processed for user `eede08f020e04282bb994b855e371433`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:36:40.114777 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-475b6572-3c1c-4e11-aa96-44bfaba96153 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:36:40.188020 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1309/2630] 10.222.0.42 () {58 vars in 1009 bytes} [Sun Aug 28 01:36:40 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:36:40.200770 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e90bbf9-f438-42dc-b92e-a6a9fa32775d service glance] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:36:40.202478 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-3e90bbf9-f438-42dc-b92e-a6a9fa32775d service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:36:40.210896 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:36:40.288269 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e90bbf9-f438-42dc-b92e-a6a9fa32775d service glance] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:36:40.299572 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a513bf60-5f45-44bf-836f-162cfaa7e522 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac4c5584582f4f55a7085af4cbbabf9b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:36:40.301421 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a513bf60-5f45-44bf-836f-162cfaa7e522 service swift] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:36:40.301947 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a513bf60-5f45-44bf-836f-162cfaa7e522 service swift] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:36:40.302429 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a513bf60-5f45-44bf-836f-162cfaa7e522 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:36:40.314250 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a513bf60-5f45-44bf-836f-162cfaa7e522 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:36:40.316725 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a513bf60-5f45-44bf-836f-162cfaa7e522 service swift] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:36:40.329544 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1322/2631] 10.222.0.42 () {60 vars in 1384 bytes} [Sun Aug 28 01:36:40 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 129 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 28 01:36:46.952096 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-475b6572-3c1c-4e11-aa96-44bfaba96153 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:36:46.953640 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-475b6572-3c1c-4e11-aa96-44bfaba96153 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:36:46.970309 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:36:47.032823 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-475b6572-3c1c-4e11-aa96-44bfaba96153 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:36:47.036840 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83e1a762-2550-451a-884b-ab7d739f0066 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:36:47.037504 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e1a762-2550-451a-884b-ab7d739f0066 service cinder] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:36:47.037818 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e1a762-2550-451a-884b-ab7d739f0066 service cinder] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:36:47.038064 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e1a762-2550-451a-884b-ab7d739f0066 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:36:47.039958 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-83e1a762-2550-451a-884b-ab7d739f0066 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:36:47.107292 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83e1a762-2550-451a-884b-ab7d739f0066 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:36:47.109206 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83e1a762-2550-451a-884b-ab7d739f0066 service cinder] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:36:47.129381 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1310/2632] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:36:46 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 178 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:36:48.644536 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78504c7e-fb1e-4543-9600-f8b172047bc2 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:36:48.645110 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78504c7e-fb1e-4543-9600-f8b172047bc2 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:36:48.645110 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78504c7e-fb1e-4543-9600-f8b172047bc2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:36:48.666513 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-78504c7e-fb1e-4543-9600-f8b172047bc2 None None] MFA Rules not processed for user `c5bc73cffe1e4fa095051e109b5f0d0c`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:36:48.667082 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-78504c7e-fb1e-4543-9600-f8b172047bc2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:36:48.674479 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1323/2633] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:36:48 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:36:48.946155 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83e1a762-2550-451a-884b-ab7d739f0066 service cinder] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:36:48.947710 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-83e1a762-2550-451a-884b-ab7d739f0066 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:36:49.016673 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83e1a762-2550-451a-884b-ab7d739f0066 service cinder] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:36:49.021599 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-921e100a-0783-4aed-b03c-3f4b20203085 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:36:49.022214 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-921e100a-0783-4aed-b03c-3f4b20203085 service neutron] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:36:49.022490 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-921e100a-0783-4aed-b03c-3f4b20203085 service neutron] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:36:49.022751 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-921e100a-0783-4aed-b03c-3f4b20203085 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:36:49.025067 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-921e100a-0783-4aed-b03c-3f4b20203085 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:36:49.036040 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-921e100a-0783-4aed-b03c-3f4b20203085 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:36:49.037495 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-921e100a-0783-4aed-b03c-3f4b20203085 service neutron] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:36:49.046608 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1311/2634] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:36:48 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:36:50.861555 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4abe001-881d-4fa2-b6a8-fc78b0b88b31 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:36:50.862264 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4abe001-881d-4fa2-b6a8-fc78b0b88b31 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:36:50.862264 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4abe001-881d-4fa2-b6a8-fc78b0b88b31 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:36:50.884588 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-d4abe001-881d-4fa2-b6a8-fc78b0b88b31 None None] MFA Rules not processed for user `eede08f020e04282bb994b855e371433`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:36:50.885932 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-d4abe001-881d-4fa2-b6a8-fc78b0b88b31 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:36:50.893792 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1324/2635] 10.222.0.42 () {58 vars in 1009 bytes} [Sun Aug 28 01:36:50 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:36:50.908432 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-921e100a-0783-4aed-b03c-3f4b20203085 service neutron] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:36:50.921831 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-921e100a-0783-4aed-b03c-3f4b20203085 service neutron] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:36:50.926998 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea10e409-85f0-4fde-b98a-5e6922f6188b service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac4c5584582f4f55a7085af4cbbabf9b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:36:50.927619 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea10e409-85f0-4fde-b98a-5e6922f6188b service swift] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:36:50.927895 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea10e409-85f0-4fde-b98a-5e6922f6188b service swift] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:36:50.928181 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea10e409-85f0-4fde-b98a-5e6922f6188b service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:36:50.937420 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea10e409-85f0-4fde-b98a-5e6922f6188b service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:36:50.938980 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea10e409-85f0-4fde-b98a-5e6922f6188b service swift] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:36:50.947643 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1312/2636] 10.222.0.42 () {60 vars in 1384 bytes} [Sun Aug 28 01:36:50 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 28 01:36:52.582842 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4abe001-881d-4fa2-b6a8-fc78b0b88b31 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:36:52.596352 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4abe001-881d-4fa2-b6a8-fc78b0b88b31 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:36:52.610918 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98382a8a-8b18-470d-a315-7e1181363050 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:36:52.611869 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98382a8a-8b18-470d-a315-7e1181363050 service neutron] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:36:52.612167 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98382a8a-8b18-470d-a315-7e1181363050 service neutron] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:36:52.612701 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98382a8a-8b18-470d-a315-7e1181363050 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:36:52.616063 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-98382a8a-8b18-470d-a315-7e1181363050 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:36:52.737671 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98382a8a-8b18-470d-a315-7e1181363050 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:36:52.739148 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98382a8a-8b18-470d-a315-7e1181363050 service neutron] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:36:52.749926 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1325/2637] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:36:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 168 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:36:56.737190 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea10e409-85f0-4fde-b98a-5e6922f6188b service swift] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:36:56.739517 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-ea10e409-85f0-4fde-b98a-5e6922f6188b service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:36:56.829128 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:36:56.832826 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98382a8a-8b18-470d-a315-7e1181363050 service neutron] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:36:56.834597 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-98382a8a-8b18-470d-a315-7e1181363050 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:36:56.855055 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98382a8a-8b18-470d-a315-7e1181363050 service neutron] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:36:56.873284 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95829d4a-04dc-4448-b116-be568593145f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:36:56.874481 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea10e409-85f0-4fde-b98a-5e6922f6188b service swift] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:36:56.875685 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95829d4a-04dc-4448-b116-be568593145f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:36:56.876184 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95829d4a-04dc-4448-b116-be568593145f admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:36:56.876688 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95829d4a-04dc-4448-b116-be568593145f admin admin] PATH_INFO: `/v3/users/3de5d68f67784e33bdcd19a9cdfed744` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:36:56.878328 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95829d4a-04dc-4448-b116-be568593145f admin admin] RBAC: Authorizing `identity:delete_user(user_id=3de5d68f67784e33bdcd19a9cdfed744)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:36:56.881665 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95829d4a-04dc-4448-b116-be568593145f admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:36:56.882322 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9a9725b-29ad-4c9a-bdab-fea8253625a3 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2a3834fa6894dada3291e4781bbd553', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:36:56.883419 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a9725b-29ad-4c9a-bdab-fea8253625a3 service placement] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:36:56.883526 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a9725b-29ad-4c9a-bdab-fea8253625a3 service placement] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:36:56.883914 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a9725b-29ad-4c9a-bdab-fea8253625a3 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:36:56.886873 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-c9a9725b-29ad-4c9a-bdab-fea8253625a3 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:36:56.902457 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9a9725b-29ad-4c9a-bdab-fea8253625a3 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:36:56.904664 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9a9725b-29ad-4c9a-bdab-fea8253625a3 service placement] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:36:56.917666 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1313/2638] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:36:56 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 181 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:36:56.943182 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-95829d4a-04dc-4448-b116-be568593145f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3de5d68f67784e33bdcd19a9cdfed744'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:36:56.943724 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-95829d4a-04dc-4448-b116-be568593145f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3de5d68f67784e33bdcd19a9cdfed744'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:36:56.953395 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-95829d4a-04dc-4448-b116-be568593145f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3de5d68f67784e33bdcd19a9cdfed744'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:36:56.960815 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-95829d4a-04dc-4448-b116-be568593145f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3de5d68f67784e33bdcd19a9cdfed744'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:36:56.961556 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-95829d4a-04dc-4448-b116-be568593145f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3de5d68f67784e33bdcd19a9cdfed744'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:36:56.967380 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:36:56.982002 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1326/2639] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 01:36:56 2022] DELETE /identity/v3/users/3de5d68f67784e33bdcd19a9cdfed744 => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 28 01:36:57.194999 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9a9725b-29ad-4c9a-bdab-fea8253625a3 service placement] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:36:57.218829 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9a9725b-29ad-4c9a-bdab-fea8253625a3 service placement] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:36:57.307463 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6d4fe60-4b4b-45fe-87f9-4b61867ac46b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:36:57.308265 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6d4fe60-4b4b-45fe-87f9-4b61867ac46b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:36:57.308548 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6d4fe60-4b4b-45fe-87f9-4b61867ac46b admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:36:57.308830 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6d4fe60-4b4b-45fe-87f9-4b61867ac46b admin admin] PATH_INFO: `/v3/projects/2348c1ddbec1476bacb194214794015d` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:36:57.310296 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6d4fe60-4b4b-45fe-87f9-4b61867ac46b admin admin] RBAC: Authorizing `identity:delete_project(project_id=2348c1ddbec1476bacb194214794015d)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:36:57.316519 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6d4fe60-4b4b-45fe-87f9-4b61867ac46b admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:36:57.396321 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-e6d4fe60-4b4b-45fe-87f9-4b61867ac46b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2348c1ddbec1476bacb194214794015d', 'request_id': 'req-e6d4fe60-4b4b-45fe-87f9-4b61867ac46b'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:36:57.402896 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-e6d4fe60-4b4b-45fe-87f9-4b61867ac46b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2348c1ddbec1476bacb194214794015d', 'request_id': 'req-e6d4fe60-4b4b-45fe-87f9-4b61867ac46b'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:36:57.409782 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-e6d4fe60-4b4b-45fe-87f9-4b61867ac46b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2348c1ddbec1476bacb194214794015d', 'request_id': 'req-e6d4fe60-4b4b-45fe-87f9-4b61867ac46b'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:36:57.428402 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-e6d4fe60-4b4b-45fe-87f9-4b61867ac46b admin admin] The token cache is being invalidate because project 2348c1ddbec1476bacb194214794015d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81507) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 01:36:57.428775 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-e6d4fe60-4b4b-45fe-87f9-4b61867ac46b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:36:57.429460 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-e6d4fe60-4b4b-45fe-87f9-4b61867ac46b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:36:57.431418 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1314/2640] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 01:36:57 2022] DELETE /identity/v3/projects/2348c1ddbec1476bacb194214794015d => generated 0 bytes in 238 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:36:57.437913 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ffd61fb-d649-4e90-91de-8e658499c792 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:36:57.438123 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ffd61fb-d649-4e90-91de-8e658499c792 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:36:57.438396 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ffd61fb-d649-4e90-91de-8e658499c792 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:36:57.459849 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-6ffd61fb-d649-4e90-91de-8e658499c792 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:36:57.461161 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-6ffd61fb-d649-4e90-91de-8e658499c792 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:36:57.633441 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1327/2641] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:36:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 197 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:36:57.715142 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f6ddfa4-c40d-4da5-ab76-3e982b6c3212 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:36:57.715376 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f6ddfa4-c40d-4da5-ab76-3e982b6c3212 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:36:57.715777 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f6ddfa4-c40d-4da5-ab76-3e982b6c3212 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:36:57.744599 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-9f6ddfa4-c40d-4da5-ab76-3e982b6c3212 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:36:57.745756 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-9f6ddfa4-c40d-4da5-ab76-3e982b6c3212 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:36:57.754131 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1315/2642] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:36:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:36:57.758505 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ffd61fb-d649-4e90-91de-8e658499c792 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:36:57.769820 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ffd61fb-d649-4e90-91de-8e658499c792 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:36:57.774893 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-186fff2a-afc6-458d-a2e2-d77daaf56698 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:36:57.775516 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-186fff2a-afc6-458d-a2e2-d77daaf56698 admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:36:57.775791 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-186fff2a-afc6-458d-a2e2-d77daaf56698 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:36:57.776057 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-186fff2a-afc6-458d-a2e2-d77daaf56698 admin admin] PATH_INFO: `/v3/domains` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:36:57.776713 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-186fff2a-afc6-458d-a2e2-d77daaf56698 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 01:36:57.777074 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-186fff2a-afc6-458d-a2e2-d77daaf56698 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:36:57.778720 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-186fff2a-afc6-458d-a2e2-d77daaf56698 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:36:57.795749 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1328/2643] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Aug 28 01:36:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 01:36:57.797271 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f6ddfa4-c40d-4da5-ab76-3e982b6c3212 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:36:57.815907 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f6ddfa4-c40d-4da5-ab76-3e982b6c3212 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:36:57.821016 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7a018d5-4779-4699-90b9-0de5367f22e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:36:57.821646 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7a018d5-4779-4699-90b9-0de5367f22e6 admin admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:36:57.821927 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7a018d5-4779-4699-90b9-0de5367f22e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:36:57.822194 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7a018d5-4779-4699-90b9-0de5367f22e6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:36:57.822886 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7a018d5-4779-4699-90b9-0de5367f22e6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:36:57.825013 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7a018d5-4779-4699-90b9-0de5367f22e6 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:36:57.882773 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1316/2644] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 01:36:57 2022] POST /identity/v3/projects => generated 373 bytes in 86 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 01:36:57.888105 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-186fff2a-afc6-458d-a2e2-d77daaf56698 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:36:57.903678 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-186fff2a-afc6-458d-a2e2-d77daaf56698 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:36:57.969209 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6429803b-aa5a-4b39-a9f3-b18f064d4f00 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:36:57.969856 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6429803b-aa5a-4b39-a9f3-b18f064d4f00 admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:36:57.970199 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6429803b-aa5a-4b39-a9f3-b18f064d4f00 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:36:57.970658 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6429803b-aa5a-4b39-a9f3-b18f064d4f00 admin admin] PATH_INFO: `/v3/users` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:36:57.971368 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6429803b-aa5a-4b39-a9f3-b18f064d4f00 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:36:57.973450 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6429803b-aa5a-4b39-a9f3-b18f064d4f00 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:36:58.000428 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1329/2645] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 01:36:57 2022] POST /identity/v3/users => generated 287 bytes in 114 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 01:36:58.005047 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7a018d5-4779-4699-90b9-0de5367f22e6 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:36:58.021520 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7a018d5-4779-4699-90b9-0de5367f22e6 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:36:58.026591 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-784d2cda-d3a6-4cda-a566-ff52c339ae60 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:36:58.027123 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-784d2cda-d3a6-4cda-a566-ff52c339ae60 admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:36:58.027401 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-784d2cda-d3a6-4cda-a566-ff52c339ae60 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:36:58.027677 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-784d2cda-d3a6-4cda-a566-ff52c339ae60 admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:36:58.028321 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-784d2cda-d3a6-4cda-a566-ff52c339ae60 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:36:58.029997 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-784d2cda-d3a6-4cda-a566-ff52c339ae60 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:36:58.041805 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1317/2646] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 01:36:58 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 28 01:36:58.046023 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6429803b-aa5a-4b39-a9f3-b18f064d4f00 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:36:58.056638 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6429803b-aa5a-4b39-a9f3-b18f064d4f00 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:36:58.061383 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0f90f37-9da8-448d-ab48-ae04809ec84f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:36:58.062008 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0f90f37-9da8-448d-ab48-ae04809ec84f admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:36:58.062282 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0f90f37-9da8-448d-ab48-ae04809ec84f admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:36:58.062545 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0f90f37-9da8-448d-ab48-ae04809ec84f admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:36:58.063178 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0f90f37-9da8-448d-ab48-ae04809ec84f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:36:58.064833 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0f90f37-9da8-448d-ab48-ae04809ec84f admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:36:58.075440 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1330/2647] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 01:36:58 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 01:36:58.079717 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-784d2cda-d3a6-4cda-a566-ff52c339ae60 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:36:58.090196 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-784d2cda-d3a6-4cda-a566-ff52c339ae60 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:36:58.095556 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb203e1f-dd31-44c0-b7d0-91dbee614b8f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:36:58.096198 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb203e1f-dd31-44c0-b7d0-91dbee614b8f admin admin] REQUEST_METHOD: `PUT` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:36:58.096480 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb203e1f-dd31-44c0-b7d0-91dbee614b8f admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:36:58.096757 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb203e1f-dd31-44c0-b7d0-91dbee614b8f admin admin] PATH_INFO: `/v3/projects/9f22397792284cdc8601883d9b7f1f43/users/382c06b725b54f09b976f8782eb0ee90/roles/0487ef7e943e4701915cb8ce0f2899d7` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:36:58.112255 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb203e1f-dd31-44c0-b7d0-91dbee614b8f admin admin] RBAC: Authorizing `identity:create_grant(project_id=9f22397792284cdc8601883d9b7f1f43, user_id=382c06b725b54f09b976f8782eb0ee90, role_id=0487ef7e943e4701915cb8ce0f2899d7)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:36:58.115008 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb203e1f-dd31-44c0-b7d0-91dbee614b8f admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:36:58.131926 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1318/2648] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 01:36:58 2022] PUT /identity/v3/projects/9f22397792284cdc8601883d9b7f1f43/users/382c06b725b54f09b976f8782eb0ee90/roles/0487ef7e943e4701915cb8ce0f2899d7 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:36:58.143237 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ad3bca0-7b99-440e-a90c-101584fd1028 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:36:58.143623 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ad3bca0-7b99-440e-a90c-101584fd1028 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:36:58.144031 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ad3bca0-7b99-440e-a90c-101584fd1028 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:36:58.176252 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-3ad3bca0-7b99-440e-a90c-101584fd1028 None None] MFA Rules not processed for user `382c06b725b54f09b976f8782eb0ee90`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:36:58.177559 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-3ad3bca0-7b99-440e-a90c-101584fd1028 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:36:58.262086 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=96,keystone:DELETE=1,keystone:INSERT=3 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:36:58.291349 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1331/2649] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:36:58 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:36:58.303015 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb203e1f-dd31-44c0-b7d0-91dbee614b8f admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:36:58.305287 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-eb203e1f-dd31-44c0-b7d0-91dbee614b8f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:36:58.313077 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:36:58.408411 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb203e1f-dd31-44c0-b7d0-91dbee614b8f admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:36:58.414323 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a260be3-6202-48fb-aefb-5cb249aba1dd service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '168b07df50c24593ba73dbde5b037aed', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:36:58.414940 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a260be3-6202-48fb-aefb-5cb249aba1dd service nova] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:36:58.415218 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a260be3-6202-48fb-aefb-5cb249aba1dd service nova] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:36:58.415492 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a260be3-6202-48fb-aefb-5cb249aba1dd service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:36:58.423398 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a260be3-6202-48fb-aefb-5cb249aba1dd service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:36:58.425030 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a260be3-6202-48fb-aefb-5cb249aba1dd service nova] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:36:58.435045 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1319/2650] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:36:58 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 133 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:37:01.353307 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ad3bca0-7b99-440e-a90c-101584fd1028 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:01.449162 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ad3bca0-7b99-440e-a90c-101584fd1028 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:01.453620 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-440b6305-9c61-42c1-a916-e5d877fa25db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:01.454320 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-440b6305-9c61-42c1-a916-e5d877fa25db admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:01.454596 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-440b6305-9c61-42c1-a916-e5d877fa25db admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:01.454865 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-440b6305-9c61-42c1-a916-e5d877fa25db admin admin] PATH_INFO: `/v3/users/382c06b725b54f09b976f8782eb0ee90` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:01.455986 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-440b6305-9c61-42c1-a916-e5d877fa25db admin admin] RBAC: Authorizing `identity:delete_user(user_id=382c06b725b54f09b976f8782eb0ee90)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:01.457972 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-440b6305-9c61-42c1-a916-e5d877fa25db admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:01.509221 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-440b6305-9c61-42c1-a916-e5d877fa25db admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '382c06b725b54f09b976f8782eb0ee90'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:01.509857 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-440b6305-9c61-42c1-a916-e5d877fa25db admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '382c06b725b54f09b976f8782eb0ee90'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:01.518192 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-440b6305-9c61-42c1-a916-e5d877fa25db admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '382c06b725b54f09b976f8782eb0ee90'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:01.525286 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-440b6305-9c61-42c1-a916-e5d877fa25db admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '382c06b725b54f09b976f8782eb0ee90'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:01.525926 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-440b6305-9c61-42c1-a916-e5d877fa25db admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '382c06b725b54f09b976f8782eb0ee90'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:01.543923 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1332/2651] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 01:37:01 2022] DELETE /identity/v3/users/382c06b725b54f09b976f8782eb0ee90 => generated 0 bytes in 192 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:37:01.548862 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b43b6b18-3287-43a0-b853-d14df088ac46 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:01.549441 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b43b6b18-3287-43a0-b853-d14df088ac46 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:01.549441 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b43b6b18-3287-43a0-b853-d14df088ac46 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:01.568976 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-b43b6b18-3287-43a0-b853-d14df088ac46 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:37:01.570241 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-b43b6b18-3287-43a0-b853-d14df088ac46 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:37:01.634214 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1320/2652] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:37:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 28 01:37:01.647522 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-440b6305-9c61-42c1-a916-e5d877fa25db admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:01.649085 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-440b6305-9c61-42c1-a916-e5d877fa25db admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:37:01.745187 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-440b6305-9c61-42c1-a916-e5d877fa25db admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:01.749203 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8aeb745-a8dc-43f8-9d62-b922387399dd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:01.749830 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8aeb745-a8dc-43f8-9d62-b922387399dd service neutron] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:01.750107 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8aeb745-a8dc-43f8-9d62-b922387399dd service neutron] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:01.750383 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8aeb745-a8dc-43f8-9d62-b922387399dd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:01.760583 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8aeb745-a8dc-43f8-9d62-b922387399dd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:01.762092 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8aeb745-a8dc-43f8-9d62-b922387399dd service neutron] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:01.789056 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1333/2653] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:37:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 134 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:37:02.084886 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b43b6b18-3287-43a0-b853-d14df088ac46 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:02.087267 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-b43b6b18-3287-43a0-b853-d14df088ac46 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:37:02.104481 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b43b6b18-3287-43a0-b853-d14df088ac46 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:02.116876 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f549b29-fcf6-4257-a3ba-c8cde8bc020b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:02.119904 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f549b29-fcf6-4257-a3ba-c8cde8bc020b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:02.120281 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f549b29-fcf6-4257-a3ba-c8cde8bc020b admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:02.121458 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f549b29-fcf6-4257-a3ba-c8cde8bc020b admin admin] PATH_INFO: `/v3/projects/9f22397792284cdc8601883d9b7f1f43` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:02.123589 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f549b29-fcf6-4257-a3ba-c8cde8bc020b admin admin] RBAC: Authorizing `identity:delete_project(project_id=9f22397792284cdc8601883d9b7f1f43)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:02.127514 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f549b29-fcf6-4257-a3ba-c8cde8bc020b admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:02.185226 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-8f549b29-fcf6-4257-a3ba-c8cde8bc020b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9f22397792284cdc8601883d9b7f1f43', 'request_id': 'req-8f549b29-fcf6-4257-a3ba-c8cde8bc020b'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:02.190846 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-8f549b29-fcf6-4257-a3ba-c8cde8bc020b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9f22397792284cdc8601883d9b7f1f43', 'request_id': 'req-8f549b29-fcf6-4257-a3ba-c8cde8bc020b'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:02.197144 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-8f549b29-fcf6-4257-a3ba-c8cde8bc020b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9f22397792284cdc8601883d9b7f1f43', 'request_id': 'req-8f549b29-fcf6-4257-a3ba-c8cde8bc020b'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:02.211413 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-8f549b29-fcf6-4257-a3ba-c8cde8bc020b admin admin] The token cache is being invalidate because project 9f22397792284cdc8601883d9b7f1f43 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81507) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 01:37:02.211741 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-8f549b29-fcf6-4257-a3ba-c8cde8bc020b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:02.212306 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-8f549b29-fcf6-4257-a3ba-c8cde8bc020b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:02.213854 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1321/2654] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 01:37:02 2022] DELETE /identity/v3/projects/9f22397792284cdc8601883d9b7f1f43 => generated 0 bytes in 130 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 28 01:37:02.218898 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45de5069-1a1a-46ee-961d-c54c5ee17fbb None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:02.219208 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45de5069-1a1a-46ee-961d-c54c5ee17fbb None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:02.219482 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45de5069-1a1a-46ee-961d-c54c5ee17fbb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:02.229368 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:37:02.240756 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-45de5069-1a1a-46ee-961d-c54c5ee17fbb None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:37:02.242396 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-45de5069-1a1a-46ee-961d-c54c5ee17fbb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:37:02.303542 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1334/2655] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:37:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 28 01:37:02.312518 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cce6073-0006-4790-ad67-a70299d0b9c9 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:02.312799 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cce6073-0006-4790-ad67-a70299d0b9c9 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:02.313104 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cce6073-0006-4790-ad67-a70299d0b9c9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:02.334208 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-4cce6073-0006-4790-ad67-a70299d0b9c9 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:37:02.335441 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-4cce6073-0006-4790-ad67-a70299d0b9c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:37:02.344907 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1322/2656] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:37:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:37:02.349312 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45de5069-1a1a-46ee-961d-c54c5ee17fbb None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:02.359130 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45de5069-1a1a-46ee-961d-c54c5ee17fbb None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:02.363921 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cc9bfe3-7b09-4fde-ad2e-accf06611065 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:02.364563 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc9bfe3-7b09-4fde-ad2e-accf06611065 admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:02.364842 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc9bfe3-7b09-4fde-ad2e-accf06611065 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:02.365139 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc9bfe3-7b09-4fde-ad2e-accf06611065 admin admin] PATH_INFO: `/v3/domains` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:02.365791 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cc9bfe3-7b09-4fde-ad2e-accf06611065 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 01:37:02.366090 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cc9bfe3-7b09-4fde-ad2e-accf06611065 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:02.367739 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cc9bfe3-7b09-4fde-ad2e-accf06611065 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:02.379945 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1335/2657] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Aug 28 01:37:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 01:37:02.384374 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cce6073-0006-4790-ad67-a70299d0b9c9 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:02.390240 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:37:02.394904 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cce6073-0006-4790-ad67-a70299d0b9c9 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:02.402309 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1733872-7d2b-42e2-ba82-8e0b8e5575f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:02.403067 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1733872-7d2b-42e2-ba82-8e0b8e5575f7 admin admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:02.403462 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1733872-7d2b-42e2-ba82-8e0b8e5575f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:02.403912 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1733872-7d2b-42e2-ba82-8e0b8e5575f7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:02.404783 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1733872-7d2b-42e2-ba82-8e0b8e5575f7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:02.406999 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1733872-7d2b-42e2-ba82-8e0b8e5575f7 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:02.438748 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1323/2658] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 01:37:02 2022] POST /identity/v3/projects => generated 405 bytes in 55 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 01:37:02.443268 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cc9bfe3-7b09-4fde-ad2e-accf06611065 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:02.453538 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cc9bfe3-7b09-4fde-ad2e-accf06611065 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:02.509157 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e45024f6-e6ff-4a42-b910-675263188eb9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:02.509783 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e45024f6-e6ff-4a42-b910-675263188eb9 admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:02.510058 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e45024f6-e6ff-4a42-b910-675263188eb9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:02.510334 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e45024f6-e6ff-4a42-b910-675263188eb9 admin admin] PATH_INFO: `/v3/users` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:02.510958 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e45024f6-e6ff-4a42-b910-675263188eb9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:02.512879 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e45024f6-e6ff-4a42-b910-675263188eb9 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:02.536095 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1336/2659] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 01:37:02 2022] POST /identity/v3/users => generated 303 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 01:37:02.540338 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1733872-7d2b-42e2-ba82-8e0b8e5575f7 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:02.550917 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1733872-7d2b-42e2-ba82-8e0b8e5575f7 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:02.555923 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-189e69bc-b40a-4a57-bbca-89ee5b57a5b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:02.556517 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-189e69bc-b40a-4a57-bbca-89ee5b57a5b1 admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:02.556790 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-189e69bc-b40a-4a57-bbca-89ee5b57a5b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:02.557092 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-189e69bc-b40a-4a57-bbca-89ee5b57a5b1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:02.557725 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-189e69bc-b40a-4a57-bbca-89ee5b57a5b1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:02.559328 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-189e69bc-b40a-4a57-bbca-89ee5b57a5b1 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:02.569466 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1324/2660] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 01:37:02 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 01:37:02.573524 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e45024f6-e6ff-4a42-b910-675263188eb9 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:02.584119 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e45024f6-e6ff-4a42-b910-675263188eb9 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:02.589344 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16530633-04ed-4dd6-bea6-40da7c703e48 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:02.589944 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16530633-04ed-4dd6-bea6-40da7c703e48 admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:02.590220 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16530633-04ed-4dd6-bea6-40da7c703e48 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:02.590504 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16530633-04ed-4dd6-bea6-40da7c703e48 admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:02.591144 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16530633-04ed-4dd6-bea6-40da7c703e48 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:02.592777 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16530633-04ed-4dd6-bea6-40da7c703e48 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:02.605412 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1337/2661] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 01:37:02 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 01:37:02.609575 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-189e69bc-b40a-4a57-bbca-89ee5b57a5b1 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:02.619725 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-189e69bc-b40a-4a57-bbca-89ee5b57a5b1 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:02.626791 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8affc2ce-1045-4ff4-a5f0-fe0a11224217 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:02.627911 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8affc2ce-1045-4ff4-a5f0-fe0a11224217 admin admin] REQUEST_METHOD: `PUT` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:02.628394 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8affc2ce-1045-4ff4-a5f0-fe0a11224217 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:02.628857 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8affc2ce-1045-4ff4-a5f0-fe0a11224217 admin admin] PATH_INFO: `/v3/projects/feee65c3789140a883bf5f651ae732c6/users/39260289ce6a49dc87ee948ee811938f/roles/0487ef7e943e4701915cb8ce0f2899d7` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:02.646915 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8affc2ce-1045-4ff4-a5f0-fe0a11224217 admin admin] RBAC: Authorizing `identity:create_grant(project_id=feee65c3789140a883bf5f651ae732c6, user_id=39260289ce6a49dc87ee948ee811938f, role_id=0487ef7e943e4701915cb8ce0f2899d7)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:02.649870 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8affc2ce-1045-4ff4-a5f0-fe0a11224217 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:02.675723 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1325/2662] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 01:37:02 2022] PUT /identity/v3/projects/feee65c3789140a883bf5f651ae732c6/users/39260289ce6a49dc87ee948ee811938f/roles/0487ef7e943e4701915cb8ce0f2899d7 => generated 0 bytes in 67 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:37:02.686422 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4909a31-220b-47dc-9cab-e23a84fc36ce None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:02.686711 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4909a31-220b-47dc-9cab-e23a84fc36ce None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:02.687013 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4909a31-220b-47dc-9cab-e23a84fc36ce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:02.717362 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-d4909a31-220b-47dc-9cab-e23a84fc36ce None None] MFA Rules not processed for user `39260289ce6a49dc87ee948ee811938f`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:37:02.718596 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-d4909a31-220b-47dc-9cab-e23a84fc36ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:37:02.783981 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:37:02.787897 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1338/2663] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:37:02 2022] POST /identity/v3/auth/tokens => generated 3282 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:37:02.794145 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8affc2ce-1045-4ff4-a5f0-fe0a11224217 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:02.859773 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8affc2ce-1045-4ff4-a5f0-fe0a11224217 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:02.864975 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b3dee7b-55ec-44af-b050-ea100125de39 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:02.865646 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b3dee7b-55ec-44af-b050-ea100125de39 admin admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:02.865930 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b3dee7b-55ec-44af-b050-ea100125de39 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:02.866247 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b3dee7b-55ec-44af-b050-ea100125de39 admin admin] PATH_INFO: `/v3/projects` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:02.866962 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b3dee7b-55ec-44af-b050-ea100125de39 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:02.869117 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b3dee7b-55ec-44af-b050-ea100125de39 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:02.900513 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1326/2664] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 01:37:02 2022] POST /identity/v3/projects => generated 403 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 01:37:02.904802 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4909a31-220b-47dc-9cab-e23a84fc36ce None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:02.916515 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4909a31-220b-47dc-9cab-e23a84fc36ce None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:02.977784 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-025eff63-a907-4011-bbcb-7a63a0e2399b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:02.978220 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-025eff63-a907-4011-bbcb-7a63a0e2399b admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:02.978503 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-025eff63-a907-4011-bbcb-7a63a0e2399b admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:02.978774 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-025eff63-a907-4011-bbcb-7a63a0e2399b admin admin] PATH_INFO: `/v3/users` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:02.979444 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-025eff63-a907-4011-bbcb-7a63a0e2399b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:02.981449 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-025eff63-a907-4011-bbcb-7a63a0e2399b admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:03.005838 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1339/2665] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 01:37:02 2022] POST /identity/v3/users => generated 308 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 01:37:03.009720 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b3dee7b-55ec-44af-b050-ea100125de39 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:03.021628 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b3dee7b-55ec-44af-b050-ea100125de39 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:03.026612 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fce234fb-47f2-4b41-9f15-56346a9b5f11 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:03.027230 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce234fb-47f2-4b41-9f15-56346a9b5f11 admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:03.027515 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce234fb-47f2-4b41-9f15-56346a9b5f11 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:03.027791 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce234fb-47f2-4b41-9f15-56346a9b5f11 admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:03.028441 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fce234fb-47f2-4b41-9f15-56346a9b5f11 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:03.030222 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fce234fb-47f2-4b41-9f15-56346a9b5f11 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:03.041977 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1327/2666] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 01:37:03 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 01:37:03.046338 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-025eff63-a907-4011-bbcb-7a63a0e2399b admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:03.058849 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-025eff63-a907-4011-bbcb-7a63a0e2399b admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:03.063705 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31cc00e2-e995-4f6e-a4ac-db168f212f84 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:03.064377 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31cc00e2-e995-4f6e-a4ac-db168f212f84 admin admin] REQUEST_METHOD: `PUT` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:03.064660 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31cc00e2-e995-4f6e-a4ac-db168f212f84 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:03.064936 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31cc00e2-e995-4f6e-a4ac-db168f212f84 admin admin] PATH_INFO: `/v3/projects/8c42c05f81464fd492db6d17dde92109/users/81f94264681742b190063480661f7352/roles/f08d6043261747e3b4763c744eef7dfb` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:03.081953 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31cc00e2-e995-4f6e-a4ac-db168f212f84 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8c42c05f81464fd492db6d17dde92109, user_id=81f94264681742b190063480661f7352, role_id=f08d6043261747e3b4763c744eef7dfb)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:03.084627 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31cc00e2-e995-4f6e-a4ac-db168f212f84 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:03.100662 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1340/2667] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 01:37:03 2022] PUT /identity/v3/projects/8c42c05f81464fd492db6d17dde92109/users/81f94264681742b190063480661f7352/roles/f08d6043261747e3b4763c744eef7dfb => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:37:03.104875 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fce234fb-47f2-4b41-9f15-56346a9b5f11 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:03.119001 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fce234fb-47f2-4b41-9f15-56346a9b5f11 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:03.182148 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d07c4f93-1171-444c-9438-827e62b2e096 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:03.182776 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d07c4f93-1171-444c-9438-827e62b2e096 admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:03.183058 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d07c4f93-1171-444c-9438-827e62b2e096 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:03.183332 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d07c4f93-1171-444c-9438-827e62b2e096 admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:03.183963 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d07c4f93-1171-444c-9438-827e62b2e096 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:03.185749 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d07c4f93-1171-444c-9438-827e62b2e096 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:03.197901 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1328/2668] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 01:37:03 2022] GET /identity/v3/roles => generated 1420 bytes in 94 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 01:37:03.202438 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31cc00e2-e995-4f6e-a4ac-db168f212f84 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:03.215388 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31cc00e2-e995-4f6e-a4ac-db168f212f84 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:03.221895 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96f1f80e-4355-483a-81fd-7ee1472885e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:03.222549 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96f1f80e-4355-483a-81fd-7ee1472885e2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:03.222824 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96f1f80e-4355-483a-81fd-7ee1472885e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:03.223090 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96f1f80e-4355-483a-81fd-7ee1472885e2 admin admin] PATH_INFO: `/v3/projects/8c42c05f81464fd492db6d17dde92109/users/81f94264681742b190063480661f7352/roles/f08d6043261747e3b4763c744eef7dfb` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:03.225055 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96f1f80e-4355-483a-81fd-7ee1472885e2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8c42c05f81464fd492db6d17dde92109, user_id=81f94264681742b190063480661f7352, role_id=f08d6043261747e3b4763c744eef7dfb)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:03.227666 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96f1f80e-4355-483a-81fd-7ee1472885e2 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:03.239482 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1341/2669] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 01:37:03 2022] PUT /identity/v3/projects/8c42c05f81464fd492db6d17dde92109/users/81f94264681742b190063480661f7352/roles/f08d6043261747e3b4763c744eef7dfb => generated 0 bytes in 38 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:37:03.249465 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc299af6-47d0-4231-aaac-e052cdbb6117 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:03.249748 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc299af6-47d0-4231-aaac-e052cdbb6117 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:03.250013 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc299af6-47d0-4231-aaac-e052cdbb6117 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:03.278570 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-dc299af6-47d0-4231-aaac-e052cdbb6117 None None] MFA Rules not processed for user `81f94264681742b190063480661f7352`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:37:03.279790 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-dc299af6-47d0-4231-aaac-e052cdbb6117 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:37:03.285708 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:37:03.363252 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1329/2670] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:37:03 2022] POST /identity/v3/auth/tokens => generated 3347 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:37:03.374611 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96f1f80e-4355-483a-81fd-7ee1472885e2 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:03.376568 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-96f1f80e-4355-483a-81fd-7ee1472885e2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:37:03.437142 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96f1f80e-4355-483a-81fd-7ee1472885e2 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:03.440868 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d521cc3-116b-48b6-b6c6-a7d8ba042a47 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:03.441522 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d521cc3-116b-48b6-b6c6-a7d8ba042a47 service cinder] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:03.441802 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d521cc3-116b-48b6-b6c6-a7d8ba042a47 service cinder] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:03.442067 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d521cc3-116b-48b6-b6c6-a7d8ba042a47 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:03.452655 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d521cc3-116b-48b6-b6c6-a7d8ba042a47 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:03.454227 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d521cc3-116b-48b6-b6c6-a7d8ba042a47 service cinder] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:03.464333 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1342/2671] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:37:03 2022] GET /identity/v3/auth/tokens => generated 3347 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:37:03.680609 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc299af6-47d0-4231-aaac-e052cdbb6117 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:03.750965 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc299af6-47d0-4231-aaac-e052cdbb6117 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:03.756075 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3440371-14b7-4983-8922-c82509c315df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:03.756740 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3440371-14b7-4983-8922-c82509c315df admin admin] REQUEST_METHOD: `DELETE` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:03.757044 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3440371-14b7-4983-8922-c82509c315df admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:03.757337 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3440371-14b7-4983-8922-c82509c315df admin admin] PATH_INFO: `/v3/users/39260289ce6a49dc87ee948ee811938f` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:03.758460 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3440371-14b7-4983-8922-c82509c315df admin admin] RBAC: Authorizing `identity:delete_user(user_id=39260289ce6a49dc87ee948ee811938f)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:03.760416 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3440371-14b7-4983-8922-c82509c315df admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:03.815033 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-d3440371-14b7-4983-8922-c82509c315df admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '39260289ce6a49dc87ee948ee811938f'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:03.827473 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-d3440371-14b7-4983-8922-c82509c315df admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '39260289ce6a49dc87ee948ee811938f'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:03.837945 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-d3440371-14b7-4983-8922-c82509c315df admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '39260289ce6a49dc87ee948ee811938f'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:03.846241 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-d3440371-14b7-4983-8922-c82509c315df admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '39260289ce6a49dc87ee948ee811938f'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:03.847219 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-d3440371-14b7-4983-8922-c82509c315df admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '39260289ce6a49dc87ee948ee811938f'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:03.856152 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1330/2672] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 01:37:03 2022] DELETE /identity/v3/users/39260289ce6a49dc87ee948ee811938f => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:37:03.862786 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4b59108-7654-43c8-887b-87ba2a083b22 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:03.863143 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4b59108-7654-43c8-887b-87ba2a083b22 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:03.863497 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4b59108-7654-43c8-887b-87ba2a083b22 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:03.886045 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-e4b59108-7654-43c8-887b-87ba2a083b22 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:37:03.887282 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-e4b59108-7654-43c8-887b-87ba2a083b22 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:37:03.940188 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:37:03.955417 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1343/2673] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:37:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:37:03.965643 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3440371-14b7-4983-8922-c82509c315df admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:03.967346 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-d3440371-14b7-4983-8922-c82509c315df admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:37:04.003562 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:37:04.039641 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3440371-14b7-4983-8922-c82509c315df admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:04.044934 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7d1bf7b-5abf-4f2d-a0f3-a5069f7c21a3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:04.045801 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7d1bf7b-5abf-4f2d-a0f3-a5069f7c21a3 service neutron] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:04.046124 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7d1bf7b-5abf-4f2d-a0f3-a5069f7c21a3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:04.046367 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7d1bf7b-5abf-4f2d-a0f3-a5069f7c21a3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:04.057352 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7d1bf7b-5abf-4f2d-a0f3-a5069f7c21a3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:04.058793 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7d1bf7b-5abf-4f2d-a0f3-a5069f7c21a3 service neutron] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:04.068505 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1331/2674] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:37:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:37:04.417587 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4b59108-7654-43c8-887b-87ba2a083b22 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:04.419182 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-e4b59108-7654-43c8-887b-87ba2a083b22 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:37:04.433520 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4b59108-7654-43c8-887b-87ba2a083b22 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:04.438940 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aac5d2b-ce9e-44bc-bdf2-dc0c900c6224 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:04.439643 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aac5d2b-ce9e-44bc-bdf2-dc0c900c6224 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:04.439921 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aac5d2b-ce9e-44bc-bdf2-dc0c900c6224 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:04.440691 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aac5d2b-ce9e-44bc-bdf2-dc0c900c6224 admin admin] PATH_INFO: `/v3/projects/feee65c3789140a883bf5f651ae732c6` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:04.441798 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aac5d2b-ce9e-44bc-bdf2-dc0c900c6224 admin admin] RBAC: Authorizing `identity:delete_project(project_id=feee65c3789140a883bf5f651ae732c6)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:04.443725 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aac5d2b-ce9e-44bc-bdf2-dc0c900c6224 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:04.495579 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-2aac5d2b-ce9e-44bc-bdf2-dc0c900c6224 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'feee65c3789140a883bf5f651ae732c6', 'request_id': 'req-2aac5d2b-ce9e-44bc-bdf2-dc0c900c6224'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:04.500901 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-2aac5d2b-ce9e-44bc-bdf2-dc0c900c6224 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'feee65c3789140a883bf5f651ae732c6', 'request_id': 'req-2aac5d2b-ce9e-44bc-bdf2-dc0c900c6224'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:04.507169 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-2aac5d2b-ce9e-44bc-bdf2-dc0c900c6224 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'feee65c3789140a883bf5f651ae732c6', 'request_id': 'req-2aac5d2b-ce9e-44bc-bdf2-dc0c900c6224'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:04.523416 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-2aac5d2b-ce9e-44bc-bdf2-dc0c900c6224 admin admin] The token cache is being invalidate because project feee65c3789140a883bf5f651ae732c6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81508) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 01:37:04.523628 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-2aac5d2b-ce9e-44bc-bdf2-dc0c900c6224 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:04.524259 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-2aac5d2b-ce9e-44bc-bdf2-dc0c900c6224 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:04.525808 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1344/2675] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 01:37:04 2022] DELETE /identity/v3/projects/feee65c3789140a883bf5f651ae732c6 => generated 0 bytes in 109 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:37:04.530104 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7d1bf7b-5abf-4f2d-a0f3-a5069f7c21a3 service neutron] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:04.540790 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7d1bf7b-5abf-4f2d-a0f3-a5069f7c21a3 service neutron] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:04.608529 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-036a566d-6c0b-471a-a79a-a85389aa1302 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:04.609210 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-036a566d-6c0b-471a-a79a-a85389aa1302 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:04.609494 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-036a566d-6c0b-471a-a79a-a85389aa1302 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:04.609768 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-036a566d-6c0b-471a-a79a-a85389aa1302 admin admin] PATH_INFO: `/v3/users/81f94264681742b190063480661f7352` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:04.611027 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-036a566d-6c0b-471a-a79a-a85389aa1302 admin admin] RBAC: Authorizing `identity:delete_user(user_id=81f94264681742b190063480661f7352)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:04.613030 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-036a566d-6c0b-471a-a79a-a85389aa1302 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:04.663212 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-036a566d-6c0b-471a-a79a-a85389aa1302 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '81f94264681742b190063480661f7352'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:04.677478 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-036a566d-6c0b-471a-a79a-a85389aa1302 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '81f94264681742b190063480661f7352'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:04.687333 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-036a566d-6c0b-471a-a79a-a85389aa1302 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '81f94264681742b190063480661f7352'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:04.695311 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-036a566d-6c0b-471a-a79a-a85389aa1302 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '81f94264681742b190063480661f7352'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:04.695990 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-036a566d-6c0b-471a-a79a-a85389aa1302 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '81f94264681742b190063480661f7352'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:04.702063 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1332/2676] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 01:37:04 2022] DELETE /identity/v3/users/81f94264681742b190063480661f7352 => generated 0 bytes in 172 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 28 01:37:04.986631 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aac5d2b-ce9e-44bc-bdf2-dc0c900c6224 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:04.988123 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-2aac5d2b-ce9e-44bc-bdf2-dc0c900c6224 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:37:05.006013 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aac5d2b-ce9e-44bc-bdf2-dc0c900c6224 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:05.066564 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-125f04e3-6d58-4b64-b308-9bb21a63470f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:05.067098 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-125f04e3-6d58-4b64-b308-9bb21a63470f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:05.067373 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-125f04e3-6d58-4b64-b308-9bb21a63470f admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:05.067642 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-125f04e3-6d58-4b64-b308-9bb21a63470f admin admin] PATH_INFO: `/v3/projects/8c42c05f81464fd492db6d17dde92109` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:05.069074 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-125f04e3-6d58-4b64-b308-9bb21a63470f admin admin] RBAC: Authorizing `identity:delete_project(project_id=8c42c05f81464fd492db6d17dde92109)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:05.070976 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-125f04e3-6d58-4b64-b308-9bb21a63470f admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:05.127717 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-125f04e3-6d58-4b64-b308-9bb21a63470f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8c42c05f81464fd492db6d17dde92109', 'request_id': 'req-125f04e3-6d58-4b64-b308-9bb21a63470f'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:05.133517 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-125f04e3-6d58-4b64-b308-9bb21a63470f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8c42c05f81464fd492db6d17dde92109', 'request_id': 'req-125f04e3-6d58-4b64-b308-9bb21a63470f'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:05.139782 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-125f04e3-6d58-4b64-b308-9bb21a63470f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8c42c05f81464fd492db6d17dde92109', 'request_id': 'req-125f04e3-6d58-4b64-b308-9bb21a63470f'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:05.153762 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-125f04e3-6d58-4b64-b308-9bb21a63470f admin admin] The token cache is being invalidate because project 8c42c05f81464fd492db6d17dde92109 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81508) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 01:37:05.154050 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-125f04e3-6d58-4b64-b308-9bb21a63470f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:05.154747 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-125f04e3-6d58-4b64-b308-9bb21a63470f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:05.156233 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1345/2677] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 01:37:04 2022] DELETE /identity/v3/projects/8c42c05f81464fd492db6d17dde92109 => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:37:05.161387 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2943b369-8437-4199-b322-0aa0489cc000 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:05.161656 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2943b369-8437-4199-b322-0aa0489cc000 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:05.161932 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2943b369-8437-4199-b322-0aa0489cc000 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:05.182336 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-2943b369-8437-4199-b322-0aa0489cc000 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:37:05.183541 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-2943b369-8437-4199-b322-0aa0489cc000 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:37:05.224052 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:37:05.244083 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1333/2678] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:37:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:37:05.253276 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6a7e250-56b3-4803-b823-cd3d56745e1c None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:05.253560 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6a7e250-56b3-4803-b823-cd3d56745e1c None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:05.253831 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6a7e250-56b3-4803-b823-cd3d56745e1c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:05.274255 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-a6a7e250-56b3-4803-b823-cd3d56745e1c None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:37:05.275483 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-a6a7e250-56b3-4803-b823-cd3d56745e1c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:37:05.284410 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1346/2679] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:37:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:37:05.288904 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2943b369-8437-4199-b322-0aa0489cc000 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:05.299531 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2943b369-8437-4199-b322-0aa0489cc000 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:05.305426 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec28bc54-eb20-41bf-921c-9e50d751ff5a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:05.306040 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec28bc54-eb20-41bf-921c-9e50d751ff5a admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:05.306314 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec28bc54-eb20-41bf-921c-9e50d751ff5a admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:05.306586 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec28bc54-eb20-41bf-921c-9e50d751ff5a admin admin] PATH_INFO: `/v3/domains` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:05.307242 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec28bc54-eb20-41bf-921c-9e50d751ff5a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 01:37:05.307542 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec28bc54-eb20-41bf-921c-9e50d751ff5a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:05.309211 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec28bc54-eb20-41bf-921c-9e50d751ff5a admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:05.322387 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1334/2680] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Aug 28 01:37:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 01:37:05.326495 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6a7e250-56b3-4803-b823-cd3d56745e1c None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:05.331432 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:37:05.337664 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6a7e250-56b3-4803-b823-cd3d56745e1c None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:05.343602 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd61020a-6c69-4b02-8da1-e539c40ae888 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:05.344467 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd61020a-6c69-4b02-8da1-e539c40ae888 admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:05.345103 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd61020a-6c69-4b02-8da1-e539c40ae888 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:05.345602 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd61020a-6c69-4b02-8da1-e539c40ae888 admin admin] PATH_INFO: `/v3/projects` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:05.346237 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd61020a-6c69-4b02-8da1-e539c40ae888 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:05.348311 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd61020a-6c69-4b02-8da1-e539c40ae888 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:05.380175 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1347/2681] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 01:37:05 2022] POST /identity/v3/projects => generated 385 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 01:37:05.384691 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec28bc54-eb20-41bf-921c-9e50d751ff5a admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:05.395040 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec28bc54-eb20-41bf-921c-9e50d751ff5a admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:05.457312 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30d20922-e3f2-4924-91ff-ce2530eeea4a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:05.458103 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30d20922-e3f2-4924-91ff-ce2530eeea4a admin admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:05.458241 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30d20922-e3f2-4924-91ff-ce2530eeea4a admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:05.458504 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30d20922-e3f2-4924-91ff-ce2530eeea4a admin admin] PATH_INFO: `/v3/users` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:05.460419 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30d20922-e3f2-4924-91ff-ce2530eeea4a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:05.461201 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30d20922-e3f2-4924-91ff-ce2530eeea4a admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:05.487717 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1335/2682] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 01:37:05 2022] POST /identity/v3/users => generated 293 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 28 01:37:05.490389 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd61020a-6c69-4b02-8da1-e539c40ae888 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:05.501323 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd61020a-6c69-4b02-8da1-e539c40ae888 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:05.511395 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aecbcae-89f9-46b8-9225-3a45ca82f8f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:05.512021 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aecbcae-89f9-46b8-9225-3a45ca82f8f6 admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:05.512298 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aecbcae-89f9-46b8-9225-3a45ca82f8f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:05.512565 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aecbcae-89f9-46b8-9225-3a45ca82f8f6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:05.513270 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7aecbcae-89f9-46b8-9225-3a45ca82f8f6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:05.514910 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7aecbcae-89f9-46b8-9225-3a45ca82f8f6 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:05.526531 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1348/2683] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 01:37:05 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 01:37:05.531374 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30d20922-e3f2-4924-91ff-ce2530eeea4a admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:05.542246 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30d20922-e3f2-4924-91ff-ce2530eeea4a admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:05.547309 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b25d9cd1-aa12-4ba5-bd18-3e366cade389 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:05.547998 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b25d9cd1-aa12-4ba5-bd18-3e366cade389 admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:05.548302 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b25d9cd1-aa12-4ba5-bd18-3e366cade389 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:05.548573 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b25d9cd1-aa12-4ba5-bd18-3e366cade389 admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:05.549290 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b25d9cd1-aa12-4ba5-bd18-3e366cade389 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:05.550911 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b25d9cd1-aa12-4ba5-bd18-3e366cade389 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:05.562445 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1336/2684] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 01:37:05 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 01:37:05.567396 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aecbcae-89f9-46b8-9225-3a45ca82f8f6 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:05.578918 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aecbcae-89f9-46b8-9225-3a45ca82f8f6 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:05.584115 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ea6531c-067a-4246-85ea-20c98a440851 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:05.584775 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ea6531c-067a-4246-85ea-20c98a440851 admin admin] REQUEST_METHOD: `PUT` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:05.585085 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ea6531c-067a-4246-85ea-20c98a440851 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:05.585359 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ea6531c-067a-4246-85ea-20c98a440851 admin admin] PATH_INFO: `/v3/projects/55a5d96c2e9d491ab9f56d51e91da4af/users/4b0509c4a852416db17a0e9d80f3a79a/roles/0487ef7e943e4701915cb8ce0f2899d7` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:05.602251 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ea6531c-067a-4246-85ea-20c98a440851 admin admin] RBAC: Authorizing `identity:create_grant(project_id=55a5d96c2e9d491ab9f56d51e91da4af, user_id=4b0509c4a852416db17a0e9d80f3a79a, role_id=0487ef7e943e4701915cb8ce0f2899d7)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:05.604923 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ea6531c-067a-4246-85ea-20c98a440851 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:05.620595 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1349/2685] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 01:37:05 2022] PUT /identity/v3/projects/55a5d96c2e9d491ab9f56d51e91da4af/users/4b0509c4a852416db17a0e9d80f3a79a/roles/0487ef7e943e4701915cb8ce0f2899d7 => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:37:05.630717 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4383e622-8073-4947-9fc2-b121291ee8bf None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:05.631004 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4383e622-8073-4947-9fc2-b121291ee8bf None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:05.631273 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4383e622-8073-4947-9fc2-b121291ee8bf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:05.660991 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-4383e622-8073-4947-9fc2-b121291ee8bf None None] MFA Rules not processed for user `4b0509c4a852416db17a0e9d80f3a79a`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:37:05.662347 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-4383e622-8073-4947-9fc2-b121291ee8bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:37:05.732734 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1337/2686] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:37:05 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:37:05.739286 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ea6531c-067a-4246-85ea-20c98a440851 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:05.811356 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ea6531c-067a-4246-85ea-20c98a440851 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:05.816307 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1111198-733f-4f66-9141-ad137617d467 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:05.816937 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1111198-733f-4f66-9141-ad137617d467 admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:05.817247 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1111198-733f-4f66-9141-ad137617d467 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:05.817521 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1111198-733f-4f66-9141-ad137617d467 admin admin] PATH_INFO: `/v3/projects` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:05.818175 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1111198-733f-4f66-9141-ad137617d467 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:05.820113 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1111198-733f-4f66-9141-ad137617d467 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:05.852406 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1350/2687] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 01:37:05 2022] POST /identity/v3/projects => generated 387 bytes in 113 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 01:37:05.856418 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4383e622-8073-4947-9fc2-b121291ee8bf None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:05.867858 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4383e622-8073-4947-9fc2-b121291ee8bf None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:05.910073 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:37:05.930930 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bd1bbf1-cbd9-46c0-8e7a-a1514d43e826 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:05.931554 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bd1bbf1-cbd9-46c0-8e7a-a1514d43e826 admin admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:05.931830 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bd1bbf1-cbd9-46c0-8e7a-a1514d43e826 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:05.932127 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bd1bbf1-cbd9-46c0-8e7a-a1514d43e826 admin admin] PATH_INFO: `/v3/users` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:05.932759 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bd1bbf1-cbd9-46c0-8e7a-a1514d43e826 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:05.934759 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bd1bbf1-cbd9-46c0-8e7a-a1514d43e826 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:05.956829 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1338/2688] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 01:37:05 2022] POST /identity/v3/users => generated 300 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 01:37:05.960883 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1111198-733f-4f66-9141-ad137617d467 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:05.971220 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1111198-733f-4f66-9141-ad137617d467 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:05.975788 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50e29d8d-bdeb-42f4-bb6d-53d3e1abb319 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:05.976388 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e29d8d-bdeb-42f4-bb6d-53d3e1abb319 admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:05.976667 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e29d8d-bdeb-42f4-bb6d-53d3e1abb319 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:05.976938 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e29d8d-bdeb-42f4-bb6d-53d3e1abb319 admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:05.977605 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50e29d8d-bdeb-42f4-bb6d-53d3e1abb319 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:05.979216 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50e29d8d-bdeb-42f4-bb6d-53d3e1abb319 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:05.989729 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1351/2689] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 01:37:05 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 01:37:05.993919 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bd1bbf1-cbd9-46c0-8e7a-a1514d43e826 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:06.007034 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bd1bbf1-cbd9-46c0-8e7a-a1514d43e826 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:06.015335 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28bf41f3-36ce-418a-928d-b1fbbd97e055 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:06.016339 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28bf41f3-36ce-418a-928d-b1fbbd97e055 admin admin] REQUEST_METHOD: `PUT` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:06.016716 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28bf41f3-36ce-418a-928d-b1fbbd97e055 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:06.017160 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28bf41f3-36ce-418a-928d-b1fbbd97e055 admin admin] PATH_INFO: `/v3/projects/72b7adc07e0846a39fb7d40190f2fb2c/users/b661c89157064c8494890588472263ca/roles/f08d6043261747e3b4763c744eef7dfb` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:06.038017 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28bf41f3-36ce-418a-928d-b1fbbd97e055 admin admin] RBAC: Authorizing `identity:create_grant(project_id=72b7adc07e0846a39fb7d40190f2fb2c, user_id=b661c89157064c8494890588472263ca, role_id=f08d6043261747e3b4763c744eef7dfb)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:06.040681 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28bf41f3-36ce-418a-928d-b1fbbd97e055 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:06.058752 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1339/2690] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 01:37:05 2022] PUT /identity/v3/projects/72b7adc07e0846a39fb7d40190f2fb2c/users/b661c89157064c8494890588472263ca/roles/f08d6043261747e3b4763c744eef7dfb => generated 0 bytes in 65 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 28 01:37:06.064146 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50e29d8d-bdeb-42f4-bb6d-53d3e1abb319 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:06.075006 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50e29d8d-bdeb-42f4-bb6d-53d3e1abb319 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:06.140027 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c89ee7c7-e1f4-4af3-b0f7-88f57018807b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:06.140635 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c89ee7c7-e1f4-4af3-b0f7-88f57018807b admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:06.140988 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c89ee7c7-e1f4-4af3-b0f7-88f57018807b admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:06.141278 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c89ee7c7-e1f4-4af3-b0f7-88f57018807b admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:06.141916 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c89ee7c7-e1f4-4af3-b0f7-88f57018807b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:06.143555 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c89ee7c7-e1f4-4af3-b0f7-88f57018807b admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:06.153639 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1352/2691] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 01:37:06 2022] GET /identity/v3/roles => generated 1420 bytes in 90 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 01:37:06.157996 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28bf41f3-36ce-418a-928d-b1fbbd97e055 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:06.169396 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28bf41f3-36ce-418a-928d-b1fbbd97e055 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:06.175733 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-976364c6-0cf5-426b-9e7a-e4cbf47d4efb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:06.176756 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-976364c6-0cf5-426b-9e7a-e4cbf47d4efb admin admin] REQUEST_METHOD: `PUT` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:06.177182 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-976364c6-0cf5-426b-9e7a-e4cbf47d4efb admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:06.177530 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-976364c6-0cf5-426b-9e7a-e4cbf47d4efb admin admin] PATH_INFO: `/v3/projects/72b7adc07e0846a39fb7d40190f2fb2c/users/b661c89157064c8494890588472263ca/roles/f08d6043261747e3b4763c744eef7dfb` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:06.181346 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-976364c6-0cf5-426b-9e7a-e4cbf47d4efb admin admin] RBAC: Authorizing `identity:create_grant(project_id=72b7adc07e0846a39fb7d40190f2fb2c, user_id=b661c89157064c8494890588472263ca, role_id=f08d6043261747e3b4763c744eef7dfb)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:06.185820 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-976364c6-0cf5-426b-9e7a-e4cbf47d4efb admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:06.198803 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1340/2692] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 01:37:06 2022] PUT /identity/v3/projects/72b7adc07e0846a39fb7d40190f2fb2c/users/b661c89157064c8494890588472263ca/roles/f08d6043261747e3b4763c744eef7dfb => generated 0 bytes in 41 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:37:06.209526 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17d3e708-0834-46db-a3ea-6212b934bfed None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:06.209843 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17d3e708-0834-46db-a3ea-6212b934bfed None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:06.210115 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17d3e708-0834-46db-a3ea-6212b934bfed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:06.240049 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-17d3e708-0834-46db-a3ea-6212b934bfed None None] MFA Rules not processed for user `b661c89157064c8494890588472263ca`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:37:06.241343 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-17d3e708-0834-46db-a3ea-6212b934bfed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:37:06.245748 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:37:06.323769 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1353/2693] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:37:06 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:37:06.338343 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-976364c6-0cf5-426b-9e7a-e4cbf47d4efb admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:06.340757 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-976364c6-0cf5-426b-9e7a-e4cbf47d4efb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:37:06.406156 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-976364c6-0cf5-426b-9e7a-e4cbf47d4efb admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:06.410106 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16cb967d-9aac-4b26-99d7-4715be56b0d2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:06.410713 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16cb967d-9aac-4b26-99d7-4715be56b0d2 service cinder] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:06.410986 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16cb967d-9aac-4b26-99d7-4715be56b0d2 service cinder] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:06.411260 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16cb967d-9aac-4b26-99d7-4715be56b0d2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:06.421492 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16cb967d-9aac-4b26-99d7-4715be56b0d2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:06.423133 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16cb967d-9aac-4b26-99d7-4715be56b0d2 service cinder] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:06.434452 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1341/2694] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:37:06 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:37:06.468877 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17d3e708-0834-46db-a3ea-6212b934bfed None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:06.534132 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17d3e708-0834-46db-a3ea-6212b934bfed None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:06.539643 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f9734f9-f2a3-419d-b068-4ae90438a28e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:06.540292 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f9734f9-f2a3-419d-b068-4ae90438a28e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:06.540573 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f9734f9-f2a3-419d-b068-4ae90438a28e admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:06.540844 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f9734f9-f2a3-419d-b068-4ae90438a28e admin admin] PATH_INFO: `/v3/users/4b0509c4a852416db17a0e9d80f3a79a` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:06.542248 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f9734f9-f2a3-419d-b068-4ae90438a28e admin admin] RBAC: Authorizing `identity:delete_user(user_id=4b0509c4a852416db17a0e9d80f3a79a)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:06.544207 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f9734f9-f2a3-419d-b068-4ae90438a28e admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:06.599486 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-1f9734f9-f2a3-419d-b068-4ae90438a28e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4b0509c4a852416db17a0e9d80f3a79a'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:06.600409 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-1f9734f9-f2a3-419d-b068-4ae90438a28e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4b0509c4a852416db17a0e9d80f3a79a'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:06.610777 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-1f9734f9-f2a3-419d-b068-4ae90438a28e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4b0509c4a852416db17a0e9d80f3a79a'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:06.618559 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-1f9734f9-f2a3-419d-b068-4ae90438a28e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4b0509c4a852416db17a0e9d80f3a79a'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:06.619260 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-1f9734f9-f2a3-419d-b068-4ae90438a28e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4b0509c4a852416db17a0e9d80f3a79a'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:06.638157 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1354/2695] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 01:37:06 2022] DELETE /identity/v3/users/4b0509c4a852416db17a0e9d80f3a79a => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:37:06.643815 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b73431de-3e4f-4800-af45-4c56fbc2d68a None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:06.644091 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b73431de-3e4f-4800-af45-4c56fbc2d68a None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:06.644382 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b73431de-3e4f-4800-af45-4c56fbc2d68a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:06.664204 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-b73431de-3e4f-4800-af45-4c56fbc2d68a None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:37:06.665468 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-b73431de-3e4f-4800-af45-4c56fbc2d68a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:37:06.729914 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1342/2696] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:37:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:37:06.740098 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f9734f9-f2a3-419d-b068-4ae90438a28e admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:06.741623 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-1f9734f9-f2a3-419d-b068-4ae90438a28e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:37:06.766005 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:37:06.809806 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f9734f9-f2a3-419d-b068-4ae90438a28e admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:06.813784 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1276959a-4345-4303-9955-15018c0568ac service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:06.814394 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1276959a-4345-4303-9955-15018c0568ac service neutron] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:06.814667 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1276959a-4345-4303-9955-15018c0568ac service neutron] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:06.814931 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1276959a-4345-4303-9955-15018c0568ac service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:06.824092 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1276959a-4345-4303-9955-15018c0568ac service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:06.825563 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1276959a-4345-4303-9955-15018c0568ac service neutron] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:06.834828 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1355/2697] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:37:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:37:07.143862 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b73431de-3e4f-4800-af45-4c56fbc2d68a None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:07.145595 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-b73431de-3e4f-4800-af45-4c56fbc2d68a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:37:07.158143 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b73431de-3e4f-4800-af45-4c56fbc2d68a None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:07.162825 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-884bfced-bdb1-407c-83a7-9109ec2231ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:07.163466 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-884bfced-bdb1-407c-83a7-9109ec2231ba admin admin] REQUEST_METHOD: `DELETE` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:07.163747 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-884bfced-bdb1-407c-83a7-9109ec2231ba admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:07.164018 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-884bfced-bdb1-407c-83a7-9109ec2231ba admin admin] PATH_INFO: `/v3/projects/55a5d96c2e9d491ab9f56d51e91da4af` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:07.165451 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-884bfced-bdb1-407c-83a7-9109ec2231ba admin admin] RBAC: Authorizing `identity:delete_project(project_id=55a5d96c2e9d491ab9f56d51e91da4af)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:07.167384 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-884bfced-bdb1-407c-83a7-9109ec2231ba admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:07.203314 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=93,keystone:INSERT=5,keystone:DELETE=2 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:37:07.217496 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-884bfced-bdb1-407c-83a7-9109ec2231ba admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '55a5d96c2e9d491ab9f56d51e91da4af', 'request_id': 'req-884bfced-bdb1-407c-83a7-9109ec2231ba'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:07.223164 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-884bfced-bdb1-407c-83a7-9109ec2231ba admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '55a5d96c2e9d491ab9f56d51e91da4af', 'request_id': 'req-884bfced-bdb1-407c-83a7-9109ec2231ba'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:07.229518 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-884bfced-bdb1-407c-83a7-9109ec2231ba admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '55a5d96c2e9d491ab9f56d51e91da4af', 'request_id': 'req-884bfced-bdb1-407c-83a7-9109ec2231ba'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:07.243243 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-884bfced-bdb1-407c-83a7-9109ec2231ba admin admin] The token cache is being invalidate because project 55a5d96c2e9d491ab9f56d51e91da4af was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81507) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 01:37:07.243589 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-884bfced-bdb1-407c-83a7-9109ec2231ba admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:07.244265 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-884bfced-bdb1-407c-83a7-9109ec2231ba admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:07.245904 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1343/2698] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 01:37:07 2022] DELETE /identity/v3/projects/55a5d96c2e9d491ab9f56d51e91da4af => generated 0 bytes in 103 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:37:07.250253 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1276959a-4345-4303-9955-15018c0568ac service neutron] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:07.260409 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1276959a-4345-4303-9955-15018c0568ac service neutron] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:07.321597 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-540b3ff3-fadb-4289-b0c4-803273f7d117 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:07.322230 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-540b3ff3-fadb-4289-b0c4-803273f7d117 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:07.322507 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-540b3ff3-fadb-4289-b0c4-803273f7d117 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:07.322777 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-540b3ff3-fadb-4289-b0c4-803273f7d117 admin admin] PATH_INFO: `/v3/users/b661c89157064c8494890588472263ca` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:07.323902 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-540b3ff3-fadb-4289-b0c4-803273f7d117 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b661c89157064c8494890588472263ca)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:07.325906 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-540b3ff3-fadb-4289-b0c4-803273f7d117 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:07.386451 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-540b3ff3-fadb-4289-b0c4-803273f7d117 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b661c89157064c8494890588472263ca'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:07.387562 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-540b3ff3-fadb-4289-b0c4-803273f7d117 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b661c89157064c8494890588472263ca'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:07.397793 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-540b3ff3-fadb-4289-b0c4-803273f7d117 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b661c89157064c8494890588472263ca'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:07.405501 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-540b3ff3-fadb-4289-b0c4-803273f7d117 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b661c89157064c8494890588472263ca'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:07.406254 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-540b3ff3-fadb-4289-b0c4-803273f7d117 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b661c89157064c8494890588472263ca'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:07.424569 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1356/2699] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 01:37:07 2022] DELETE /identity/v3/users/b661c89157064c8494890588472263ca => generated 0 bytes in 175 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:37:07.721369 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-884bfced-bdb1-407c-83a7-9109ec2231ba admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:07.722826 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-884bfced-bdb1-407c-83a7-9109ec2231ba admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:37:07.736869 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-884bfced-bdb1-407c-83a7-9109ec2231ba admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:07.805250 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c32cf504-1b45-4a46-96c6-c75ba115bef9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:07.805803 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c32cf504-1b45-4a46-96c6-c75ba115bef9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:07.806083 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c32cf504-1b45-4a46-96c6-c75ba115bef9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:07.806357 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c32cf504-1b45-4a46-96c6-c75ba115bef9 admin admin] PATH_INFO: `/v3/projects/72b7adc07e0846a39fb7d40190f2fb2c` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:07.807606 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c32cf504-1b45-4a46-96c6-c75ba115bef9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=72b7adc07e0846a39fb7d40190f2fb2c)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:07.809790 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c32cf504-1b45-4a46-96c6-c75ba115bef9 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:07.865571 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-c32cf504-1b45-4a46-96c6-c75ba115bef9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '72b7adc07e0846a39fb7d40190f2fb2c', 'request_id': 'req-c32cf504-1b45-4a46-96c6-c75ba115bef9'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:07.872221 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-c32cf504-1b45-4a46-96c6-c75ba115bef9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '72b7adc07e0846a39fb7d40190f2fb2c', 'request_id': 'req-c32cf504-1b45-4a46-96c6-c75ba115bef9'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:07.878947 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-c32cf504-1b45-4a46-96c6-c75ba115bef9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '72b7adc07e0846a39fb7d40190f2fb2c', 'request_id': 'req-c32cf504-1b45-4a46-96c6-c75ba115bef9'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:07.899886 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-c32cf504-1b45-4a46-96c6-c75ba115bef9 admin admin] The token cache is being invalidate because project 72b7adc07e0846a39fb7d40190f2fb2c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81507) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 01:37:07.899886 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-c32cf504-1b45-4a46-96c6-c75ba115bef9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:07.900669 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-c32cf504-1b45-4a46-96c6-c75ba115bef9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:07.902661 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1344/2700] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 01:37:07 2022] DELETE /identity/v3/projects/72b7adc07e0846a39fb7d40190f2fb2c => generated 0 bytes in 182 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:37:07.907862 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3a9a4b0-d8a1-4462-a13f-f67d820f8658 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:07.908415 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3a9a4b0-d8a1-4462-a13f-f67d820f8658 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:07.908415 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3a9a4b0-d8a1-4462-a13f-f67d820f8658 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:07.928665 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-b3a9a4b0-d8a1-4462-a13f-f67d820f8658 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:37:07.930370 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-b3a9a4b0-d8a1-4462-a13f-f67d820f8658 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:37:07.975303 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:37:08.008840 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1357/2701] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:37:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:37:08.011084 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8769368-64f6-44ea-a405-41cbd2c7dc14 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:08.011370 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8769368-64f6-44ea-a405-41cbd2c7dc14 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:08.011641 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8769368-64f6-44ea-a405-41cbd2c7dc14 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:08.034418 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-a8769368-64f6-44ea-a405-41cbd2c7dc14 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:37:08.035675 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-a8769368-64f6-44ea-a405-41cbd2c7dc14 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:37:08.045702 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1345/2702] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:37:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:37:08.050211 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3a9a4b0-d8a1-4462-a13f-f67d820f8658 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:08.062693 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3a9a4b0-d8a1-4462-a13f-f67d820f8658 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:08.068025 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f319d6ab-3e66-4119-b853-4e92c9cf1572 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:08.068603 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f319d6ab-3e66-4119-b853-4e92c9cf1572 admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:08.068917 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f319d6ab-3e66-4119-b853-4e92c9cf1572 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:08.069234 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f319d6ab-3e66-4119-b853-4e92c9cf1572 admin admin] PATH_INFO: `/v3/domains` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:08.069899 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f319d6ab-3e66-4119-b853-4e92c9cf1572 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 01:37:08.070199 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f319d6ab-3e66-4119-b853-4e92c9cf1572 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:08.071826 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f319d6ab-3e66-4119-b853-4e92c9cf1572 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:08.087251 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1358/2703] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Aug 28 01:37:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 01:37:08.092037 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8769368-64f6-44ea-a405-41cbd2c7dc14 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:08.102315 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8769368-64f6-44ea-a405-41cbd2c7dc14 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:08.107747 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c086bd3-faab-4521-b475-656a1a5807f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:08.108423 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c086bd3-faab-4521-b475-656a1a5807f2 admin admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:08.108883 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c086bd3-faab-4521-b475-656a1a5807f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:08.109704 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c086bd3-faab-4521-b475-656a1a5807f2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:08.110401 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c086bd3-faab-4521-b475-656a1a5807f2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:08.112374 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c086bd3-faab-4521-b475-656a1a5807f2 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:08.144429 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1346/2704] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 01:37:08 2022] POST /identity/v3/projects => generated 393 bytes in 53 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 01:37:08.148592 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f319d6ab-3e66-4119-b853-4e92c9cf1572 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:08.159345 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f319d6ab-3e66-4119-b853-4e92c9cf1572 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:08.229468 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f35c930d-3413-4e24-8106-56921c42586e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:08.230267 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f35c930d-3413-4e24-8106-56921c42586e admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:08.230549 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f35c930d-3413-4e24-8106-56921c42586e admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:08.230851 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f35c930d-3413-4e24-8106-56921c42586e admin admin] PATH_INFO: `/v3/users` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:08.231470 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f35c930d-3413-4e24-8106-56921c42586e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:08.233492 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f35c930d-3413-4e24-8106-56921c42586e admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:08.258221 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1359/2705] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 01:37:08 2022] POST /identity/v3/users => generated 297 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 01:37:08.262402 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c086bd3-faab-4521-b475-656a1a5807f2 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:08.273186 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c086bd3-faab-4521-b475-656a1a5807f2 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:08.278430 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4562e1f8-690a-481a-bd1c-48d555e68ae0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:08.279068 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4562e1f8-690a-481a-bd1c-48d555e68ae0 admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:08.279345 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4562e1f8-690a-481a-bd1c-48d555e68ae0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:08.279612 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4562e1f8-690a-481a-bd1c-48d555e68ae0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:08.280249 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4562e1f8-690a-481a-bd1c-48d555e68ae0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:08.282371 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4562e1f8-690a-481a-bd1c-48d555e68ae0 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:08.292340 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1347/2706] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 01:37:08 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 01:37:08.296254 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f35c930d-3413-4e24-8106-56921c42586e admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:08.307399 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f35c930d-3413-4e24-8106-56921c42586e admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:08.312029 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c925bcef-1514-4318-a925-aa54d2ef8e21 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:08.312639 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c925bcef-1514-4318-a925-aa54d2ef8e21 admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:08.312920 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c925bcef-1514-4318-a925-aa54d2ef8e21 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:08.313256 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c925bcef-1514-4318-a925-aa54d2ef8e21 admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:08.313925 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c925bcef-1514-4318-a925-aa54d2ef8e21 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:08.315536 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c925bcef-1514-4318-a925-aa54d2ef8e21 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:08.326525 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1360/2707] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 01:37:08 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 28 01:37:08.330811 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4562e1f8-690a-481a-bd1c-48d555e68ae0 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:08.340797 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4562e1f8-690a-481a-bd1c-48d555e68ae0 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:08.345466 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c43c9ec-8613-408c-9e73-c740286cde4d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:08.346118 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c43c9ec-8613-408c-9e73-c740286cde4d admin admin] REQUEST_METHOD: `PUT` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:08.346393 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c43c9ec-8613-408c-9e73-c740286cde4d admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:08.346664 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c43c9ec-8613-408c-9e73-c740286cde4d admin admin] PATH_INFO: `/v3/projects/e0b6ee20988a4966b1257f8da6e60f6b/users/9fed10d13bdc4fc4aebf3643d2cd3f2f/roles/0487ef7e943e4701915cb8ce0f2899d7` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:08.349701 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=85,keystone:DELETE=12,keystone:INSERT=3 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:37:08.362494 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c43c9ec-8613-408c-9e73-c740286cde4d admin admin] RBAC: Authorizing `identity:create_grant(project_id=e0b6ee20988a4966b1257f8da6e60f6b, user_id=9fed10d13bdc4fc4aebf3643d2cd3f2f, role_id=0487ef7e943e4701915cb8ce0f2899d7)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:08.365577 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c43c9ec-8613-408c-9e73-c740286cde4d admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:08.383029 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1348/2708] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 01:37:08 2022] PUT /identity/v3/projects/e0b6ee20988a4966b1257f8da6e60f6b/users/9fed10d13bdc4fc4aebf3643d2cd3f2f/roles/0487ef7e943e4701915cb8ce0f2899d7 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:37:08.398520 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fa477c3-9601-42dc-8d26-768ca3584dd6 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:08.398811 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fa477c3-9601-42dc-8d26-768ca3584dd6 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:08.399112 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fa477c3-9601-42dc-8d26-768ca3584dd6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:08.431153 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-6fa477c3-9601-42dc-8d26-768ca3584dd6 None None] MFA Rules not processed for user `9fed10d13bdc4fc4aebf3643d2cd3f2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:37:08.432396 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-6fa477c3-9601-42dc-8d26-768ca3584dd6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:37:08.503556 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1361/2709] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:37:08 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:37:08.508293 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c43c9ec-8613-408c-9e73-c740286cde4d admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:08.577547 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c43c9ec-8613-408c-9e73-c740286cde4d admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:08.590015 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29ac6f3c-6dfa-439f-b2b1-ab7621e1515c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:08.590665 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ac6f3c-6dfa-439f-b2b1-ab7621e1515c admin admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:08.590940 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ac6f3c-6dfa-439f-b2b1-ab7621e1515c admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:08.591213 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ac6f3c-6dfa-439f-b2b1-ab7621e1515c admin admin] PATH_INFO: `/v3/projects` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:08.591862 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29ac6f3c-6dfa-439f-b2b1-ab7621e1515c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:08.593833 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29ac6f3c-6dfa-439f-b2b1-ab7621e1515c admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:08.625998 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1349/2710] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 01:37:08 2022] POST /identity/v3/projects => generated 393 bytes in 115 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 01:37:08.627238 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fa477c3-9601-42dc-8d26-768ca3584dd6 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:08.639439 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fa477c3-9601-42dc-8d26-768ca3584dd6 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:08.678203 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:37:08.706105 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acb17257-821c-4c4b-a85c-0838ed655cb1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:08.706724 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acb17257-821c-4c4b-a85c-0838ed655cb1 admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:08.707318 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acb17257-821c-4c4b-a85c-0838ed655cb1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:08.707318 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acb17257-821c-4c4b-a85c-0838ed655cb1 admin admin] PATH_INFO: `/v3/users` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:08.707885 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acb17257-821c-4c4b-a85c-0838ed655cb1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:08.709988 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acb17257-821c-4c4b-a85c-0838ed655cb1 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:08.735047 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1362/2711] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 01:37:08 2022] POST /identity/v3/users => generated 303 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 01:37:08.740237 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29ac6f3c-6dfa-439f-b2b1-ab7621e1515c admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:08.752335 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29ac6f3c-6dfa-439f-b2b1-ab7621e1515c admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:08.757867 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da5d5c01-e19e-488d-b008-ffbd2751a345 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:08.758611 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da5d5c01-e19e-488d-b008-ffbd2751a345 admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:08.758979 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da5d5c01-e19e-488d-b008-ffbd2751a345 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:08.759351 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da5d5c01-e19e-488d-b008-ffbd2751a345 admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:08.760111 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da5d5c01-e19e-488d-b008-ffbd2751a345 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:08.762062 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da5d5c01-e19e-488d-b008-ffbd2751a345 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:08.774115 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1350/2712] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 01:37:08 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 01:37:08.778305 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acb17257-821c-4c4b-a85c-0838ed655cb1 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:08.789244 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acb17257-821c-4c4b-a85c-0838ed655cb1 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:08.794615 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f88c3092-df30-4ca3-ba46-8df75ec776c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:08.795284 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f88c3092-df30-4ca3-ba46-8df75ec776c4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:08.795589 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f88c3092-df30-4ca3-ba46-8df75ec776c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:08.795863 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f88c3092-df30-4ca3-ba46-8df75ec776c4 admin admin] PATH_INFO: `/v3/projects/b49d07d7ba554898965350dfa0562838/users/30c8d8090ad941f580945e6077c98db0/roles/f08d6043261747e3b4763c744eef7dfb` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:08.813597 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f88c3092-df30-4ca3-ba46-8df75ec776c4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b49d07d7ba554898965350dfa0562838, user_id=30c8d8090ad941f580945e6077c98db0, role_id=f08d6043261747e3b4763c744eef7dfb)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:08.816406 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f88c3092-df30-4ca3-ba46-8df75ec776c4 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:08.832408 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1363/2713] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 01:37:08 2022] PUT /identity/v3/projects/b49d07d7ba554898965350dfa0562838/users/30c8d8090ad941f580945e6077c98db0/roles/f08d6043261747e3b4763c744eef7dfb => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:37:08.836542 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da5d5c01-e19e-488d-b008-ffbd2751a345 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:08.847917 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da5d5c01-e19e-488d-b008-ffbd2751a345 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:08.911633 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4f4619c-053b-4889-a273-eef37da632a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:08.912260 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4f4619c-053b-4889-a273-eef37da632a3 admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:08.912539 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4f4619c-053b-4889-a273-eef37da632a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:08.912802 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4f4619c-053b-4889-a273-eef37da632a3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:08.913477 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4f4619c-053b-4889-a273-eef37da632a3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:08.915097 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4f4619c-053b-4889-a273-eef37da632a3 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:08.925893 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1351/2714] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 01:37:08 2022] GET /identity/v3/roles => generated 1420 bytes in 90 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 01:37:08.930732 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f88c3092-df30-4ca3-ba46-8df75ec776c4 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:08.942138 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f88c3092-df30-4ca3-ba46-8df75ec776c4 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:08.947513 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae9a6c8a-a527-46cf-8e1c-018fa050e2ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:08.948192 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae9a6c8a-a527-46cf-8e1c-018fa050e2ea admin admin] REQUEST_METHOD: `PUT` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:08.948470 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae9a6c8a-a527-46cf-8e1c-018fa050e2ea admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:08.948743 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae9a6c8a-a527-46cf-8e1c-018fa050e2ea admin admin] PATH_INFO: `/v3/projects/b49d07d7ba554898965350dfa0562838/users/30c8d8090ad941f580945e6077c98db0/roles/f08d6043261747e3b4763c744eef7dfb` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:08.950862 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae9a6c8a-a527-46cf-8e1c-018fa050e2ea admin admin] RBAC: Authorizing `identity:create_grant(project_id=b49d07d7ba554898965350dfa0562838, user_id=30c8d8090ad941f580945e6077c98db0, role_id=f08d6043261747e3b4763c744eef7dfb)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:08.953551 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae9a6c8a-a527-46cf-8e1c-018fa050e2ea admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:08.964982 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1364/2715] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 01:37:08 2022] PUT /identity/v3/projects/b49d07d7ba554898965350dfa0562838/users/30c8d8090ad941f580945e6077c98db0/roles/f08d6043261747e3b4763c744eef7dfb => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:37:08.977337 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6760bfa6-d1da-4def-b47b-aabc179f546e None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:08.977680 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6760bfa6-d1da-4def-b47b-aabc179f546e None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:08.977919 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6760bfa6-d1da-4def-b47b-aabc179f546e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:09.015219 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-6760bfa6-d1da-4def-b47b-aabc179f546e None None] MFA Rules not processed for user `30c8d8090ad941f580945e6077c98db0`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:37:09.016624 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-6760bfa6-d1da-4def-b47b-aabc179f546e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:37:09.072770 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:37:09.102012 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1352/2716] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:37:08 2022] POST /identity/v3/auth/tokens => generated 3337 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:37:09.114263 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae9a6c8a-a527-46cf-8e1c-018fa050e2ea admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:09.116059 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-ae9a6c8a-a527-46cf-8e1c-018fa050e2ea admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:37:09.181000 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae9a6c8a-a527-46cf-8e1c-018fa050e2ea admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:09.184992 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0913744-fdb1-4f59-b461-a7d5638ea109 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:09.185620 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0913744-fdb1-4f59-b461-a7d5638ea109 service cinder] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:09.185893 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0913744-fdb1-4f59-b461-a7d5638ea109 service cinder] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:09.186159 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0913744-fdb1-4f59-b461-a7d5638ea109 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:09.196082 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0913744-fdb1-4f59-b461-a7d5638ea109 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:09.197585 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0913744-fdb1-4f59-b461-a7d5638ea109 service cinder] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:09.207626 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1365/2717] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:37:09 2022] GET /identity/v3/auth/tokens => generated 3337 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:37:09.257134 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6760bfa6-d1da-4def-b47b-aabc179f546e None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:09.258592 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-6760bfa6-d1da-4def-b47b-aabc179f546e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:37:09.272039 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6760bfa6-d1da-4def-b47b-aabc179f546e None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:09.276518 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1f13476-e4c8-4a6c-a2cb-588c2afdd336 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:09.277154 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1f13476-e4c8-4a6c-a2cb-588c2afdd336 service cinder] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:09.277431 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1f13476-e4c8-4a6c-a2cb-588c2afdd336 service cinder] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:09.277700 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1f13476-e4c8-4a6c-a2cb-588c2afdd336 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:09.340821 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1f13476-e4c8-4a6c-a2cb-588c2afdd336 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:09.342498 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1f13476-e4c8-4a6c-a2cb-588c2afdd336 service cinder] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:09.353894 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1353/2718] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:37:09 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:37:12.775472 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0913744-fdb1-4f59-b461-a7d5638ea109 service cinder] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:12.841727 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0913744-fdb1-4f59-b461-a7d5638ea109 service cinder] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:12.846998 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0766f1c-e600-4096-89bb-1c9adc08d8d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:12.847640 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0766f1c-e600-4096-89bb-1c9adc08d8d3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:12.847918 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0766f1c-e600-4096-89bb-1c9adc08d8d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:12.848188 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0766f1c-e600-4096-89bb-1c9adc08d8d3 admin admin] PATH_INFO: `/v3/users/9fed10d13bdc4fc4aebf3643d2cd3f2f` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:12.849408 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0766f1c-e600-4096-89bb-1c9adc08d8d3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9fed10d13bdc4fc4aebf3643d2cd3f2f)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:12.851343 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0766f1c-e600-4096-89bb-1c9adc08d8d3 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:12.895725 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=90,keystone:INSERT=5,keystone:DELETE=5 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:37:12.909362 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-b0766f1c-e600-4096-89bb-1c9adc08d8d3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9fed10d13bdc4fc4aebf3643d2cd3f2f'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:12.910529 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-b0766f1c-e600-4096-89bb-1c9adc08d8d3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9fed10d13bdc4fc4aebf3643d2cd3f2f'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:12.921329 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-b0766f1c-e600-4096-89bb-1c9adc08d8d3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9fed10d13bdc4fc4aebf3643d2cd3f2f'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:12.930266 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-b0766f1c-e600-4096-89bb-1c9adc08d8d3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9fed10d13bdc4fc4aebf3643d2cd3f2f'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:12.931029 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-b0766f1c-e600-4096-89bb-1c9adc08d8d3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9fed10d13bdc4fc4aebf3643d2cd3f2f'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:12.954031 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1366/2719] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 01:37:12 2022] DELETE /identity/v3/users/9fed10d13bdc4fc4aebf3643d2cd3f2f => generated 0 bytes in 179 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:37:12.960753 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18437184-18a7-455d-ac43-a937b560b749 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:12.961355 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18437184-18a7-455d-ac43-a937b560b749 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:12.961355 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18437184-18a7-455d-ac43-a937b560b749 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:12.982580 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-18437184-18a7-455d-ac43-a937b560b749 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:37:12.983835 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-18437184-18a7-455d-ac43-a937b560b749 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:37:13.059856 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1354/2720] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:37:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:37:13.071196 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0766f1c-e600-4096-89bb-1c9adc08d8d3 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:13.072629 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-b0766f1c-e600-4096-89bb-1c9adc08d8d3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:37:13.140138 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0766f1c-e600-4096-89bb-1c9adc08d8d3 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:13.144022 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c21b89a-f581-4429-9cb7-6c199ede3429 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:13.144687 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c21b89a-f581-4429-9cb7-6c199ede3429 service neutron] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:13.144997 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c21b89a-f581-4429-9cb7-6c199ede3429 service neutron] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:13.145249 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c21b89a-f581-4429-9cb7-6c199ede3429 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:13.155290 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c21b89a-f581-4429-9cb7-6c199ede3429 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:13.156860 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c21b89a-f581-4429-9cb7-6c199ede3429 service neutron] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:13.166665 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1367/2721] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:37:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:37:13.448213 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18437184-18a7-455d-ac43-a937b560b749 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:13.449843 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-18437184-18a7-455d-ac43-a937b560b749 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:37:13.465014 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18437184-18a7-455d-ac43-a937b560b749 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:13.472967 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acced1d6-90a1-42e4-bada-fc7d6dcef338 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:13.473704 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acced1d6-90a1-42e4-bada-fc7d6dcef338 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:13.473910 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acced1d6-90a1-42e4-bada-fc7d6dcef338 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:13.474163 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acced1d6-90a1-42e4-bada-fc7d6dcef338 admin admin] PATH_INFO: `/v3/projects/e0b6ee20988a4966b1257f8da6e60f6b` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:13.475460 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acced1d6-90a1-42e4-bada-fc7d6dcef338 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e0b6ee20988a4966b1257f8da6e60f6b)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:13.477438 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acced1d6-90a1-42e4-bada-fc7d6dcef338 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:13.526852 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-acced1d6-90a1-42e4-bada-fc7d6dcef338 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e0b6ee20988a4966b1257f8da6e60f6b', 'request_id': 'req-acced1d6-90a1-42e4-bada-fc7d6dcef338'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:13.532522 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-acced1d6-90a1-42e4-bada-fc7d6dcef338 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e0b6ee20988a4966b1257f8da6e60f6b', 'request_id': 'req-acced1d6-90a1-42e4-bada-fc7d6dcef338'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:13.539954 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-acced1d6-90a1-42e4-bada-fc7d6dcef338 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e0b6ee20988a4966b1257f8da6e60f6b', 'request_id': 'req-acced1d6-90a1-42e4-bada-fc7d6dcef338'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:13.557149 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-acced1d6-90a1-42e4-bada-fc7d6dcef338 admin admin] The token cache is being invalidate because project e0b6ee20988a4966b1257f8da6e60f6b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81507) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 01:37:13.557471 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-acced1d6-90a1-42e4-bada-fc7d6dcef338 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:13.558560 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-acced1d6-90a1-42e4-bada-fc7d6dcef338 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:13.560563 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1355/2722] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 01:37:13 2022] DELETE /identity/v3/projects/e0b6ee20988a4966b1257f8da6e60f6b => generated 0 bytes in 113 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:37:13.564829 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c21b89a-f581-4429-9cb7-6c199ede3429 service neutron] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:13.575186 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c21b89a-f581-4429-9cb7-6c199ede3429 service neutron] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:13.641558 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6eddfe3-f2e3-480f-9057-df46c1898c57 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:13.642198 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6eddfe3-f2e3-480f-9057-df46c1898c57 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:13.642476 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6eddfe3-f2e3-480f-9057-df46c1898c57 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:13.642734 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6eddfe3-f2e3-480f-9057-df46c1898c57 admin admin] PATH_INFO: `/v3/users/30c8d8090ad941f580945e6077c98db0` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:13.643889 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6eddfe3-f2e3-480f-9057-df46c1898c57 admin admin] RBAC: Authorizing `identity:delete_user(user_id=30c8d8090ad941f580945e6077c98db0)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:13.645873 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6eddfe3-f2e3-480f-9057-df46c1898c57 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:13.703823 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-c6eddfe3-f2e3-480f-9057-df46c1898c57 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '30c8d8090ad941f580945e6077c98db0'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:13.704478 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-c6eddfe3-f2e3-480f-9057-df46c1898c57 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '30c8d8090ad941f580945e6077c98db0'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:13.713721 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-c6eddfe3-f2e3-480f-9057-df46c1898c57 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '30c8d8090ad941f580945e6077c98db0'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:13.716130 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=85,keystone:DELETE=13,keystone:INSERT=2 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:37:13.722471 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-c6eddfe3-f2e3-480f-9057-df46c1898c57 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '30c8d8090ad941f580945e6077c98db0'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:13.723268 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-c6eddfe3-f2e3-480f-9057-df46c1898c57 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '30c8d8090ad941f580945e6077c98db0'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:13.745246 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1368/2723] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 01:37:13 2022] DELETE /identity/v3/users/30c8d8090ad941f580945e6077c98db0 => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:37:14.036340 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acced1d6-90a1-42e4-bada-fc7d6dcef338 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:14.038737 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-acced1d6-90a1-42e4-bada-fc7d6dcef338 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:37:14.046952 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:37:14.053874 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acced1d6-90a1-42e4-bada-fc7d6dcef338 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:14.139487 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9a503d2-02ff-4042-9562-c86bd91fd94c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:14.140142 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a503d2-02ff-4042-9562-c86bd91fd94c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:14.140600 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a503d2-02ff-4042-9562-c86bd91fd94c admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:14.140883 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a503d2-02ff-4042-9562-c86bd91fd94c admin admin] PATH_INFO: `/v3/projects/b49d07d7ba554898965350dfa0562838` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:14.142401 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9a503d2-02ff-4042-9562-c86bd91fd94c admin admin] RBAC: Authorizing `identity:delete_project(project_id=b49d07d7ba554898965350dfa0562838)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:14.145173 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9a503d2-02ff-4042-9562-c86bd91fd94c admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:14.210275 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-a9a503d2-02ff-4042-9562-c86bd91fd94c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b49d07d7ba554898965350dfa0562838', 'request_id': 'req-a9a503d2-02ff-4042-9562-c86bd91fd94c'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:14.216348 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-a9a503d2-02ff-4042-9562-c86bd91fd94c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b49d07d7ba554898965350dfa0562838', 'request_id': 'req-a9a503d2-02ff-4042-9562-c86bd91fd94c'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:14.223170 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-a9a503d2-02ff-4042-9562-c86bd91fd94c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b49d07d7ba554898965350dfa0562838', 'request_id': 'req-a9a503d2-02ff-4042-9562-c86bd91fd94c'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:14.245195 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-a9a503d2-02ff-4042-9562-c86bd91fd94c admin admin] The token cache is being invalidate because project b49d07d7ba554898965350dfa0562838 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81507) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 01:37:14.245673 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-a9a503d2-02ff-4042-9562-c86bd91fd94c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:14.246314 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-a9a503d2-02ff-4042-9562-c86bd91fd94c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:14.248635 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1356/2724] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 01:37:14 2022] DELETE /identity/v3/projects/b49d07d7ba554898965350dfa0562838 => generated 0 bytes in 212 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:37:14.252988 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5c7c55d-e1cd-44d1-8c07-b21a9ad20ade None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:14.253414 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5c7c55d-e1cd-44d1-8c07-b21a9ad20ade None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:14.253747 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5c7c55d-e1cd-44d1-8c07-b21a9ad20ade None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:14.275884 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-e5c7c55d-e1cd-44d1-8c07-b21a9ad20ade None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:37:14.277197 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-e5c7c55d-e1cd-44d1-8c07-b21a9ad20ade None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:37:14.349303 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1369/2725] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:37:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:37:14.356944 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-183f2737-d9db-42cd-be9e-032de78a1845 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:14.357463 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-183f2737-d9db-42cd-be9e-032de78a1845 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:14.357674 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-183f2737-d9db-42cd-be9e-032de78a1845 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:14.386438 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-183f2737-d9db-42cd-be9e-032de78a1845 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:37:14.388573 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-183f2737-d9db-42cd-be9e-032de78a1845 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:37:14.399391 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1357/2726] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:37:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:37:14.404307 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5c7c55d-e1cd-44d1-8c07-b21a9ad20ade None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:14.414801 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5c7c55d-e1cd-44d1-8c07-b21a9ad20ade None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:14.419966 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bda3421-2525-447e-a6a4-64d5958f3e1d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:14.420166 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bda3421-2525-447e-a6a4-64d5958f3e1d admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:14.420408 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bda3421-2525-447e-a6a4-64d5958f3e1d admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:14.425308 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bda3421-2525-447e-a6a4-64d5958f3e1d admin admin] PATH_INFO: `/v3/domains` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:14.426055 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bda3421-2525-447e-a6a4-64d5958f3e1d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 01:37:14.426828 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bda3421-2525-447e-a6a4-64d5958f3e1d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:14.428012 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bda3421-2525-447e-a6a4-64d5958f3e1d admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:14.445699 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1370/2727] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Aug 28 01:37:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 01:37:14.447235 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-183f2737-d9db-42cd-be9e-032de78a1845 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:14.457772 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-183f2737-d9db-42cd-be9e-032de78a1845 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:14.464798 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39239241-9186-45c3-a5dc-a6a71f531a9d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:14.465656 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39239241-9186-45c3-a5dc-a6a71f531a9d admin admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:14.465892 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39239241-9186-45c3-a5dc-a6a71f531a9d admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:14.466267 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39239241-9186-45c3-a5dc-a6a71f531a9d admin admin] PATH_INFO: `/v3/projects` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:14.467025 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39239241-9186-45c3-a5dc-a6a71f531a9d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:14.469803 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39239241-9186-45c3-a5dc-a6a71f531a9d admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:14.510934 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1358/2728] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 01:37:14 2022] POST /identity/v3/projects => generated 401 bytes in 64 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 01:37:14.515287 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bda3421-2525-447e-a6a4-64d5958f3e1d admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:14.526581 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bda3421-2525-447e-a6a4-64d5958f3e1d admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:14.599203 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e5555bd-f45b-43fa-b461-4ef42b323649 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:14.600072 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e5555bd-f45b-43fa-b461-4ef42b323649 admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:14.600420 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e5555bd-f45b-43fa-b461-4ef42b323649 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:14.600687 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e5555bd-f45b-43fa-b461-4ef42b323649 admin admin] PATH_INFO: `/v3/users` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:14.601372 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e5555bd-f45b-43fa-b461-4ef42b323649 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:14.603347 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e5555bd-f45b-43fa-b461-4ef42b323649 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:14.632572 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1371/2729] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 01:37:14 2022] POST /identity/v3/users => generated 301 bytes in 115 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 01:37:14.633838 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39239241-9186-45c3-a5dc-a6a71f531a9d admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:14.658013 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39239241-9186-45c3-a5dc-a6a71f531a9d admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:14.665259 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8ee5b3a-974c-4a24-bc51-169717ea55ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:14.665901 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8ee5b3a-974c-4a24-bc51-169717ea55ca admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:14.666196 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8ee5b3a-974c-4a24-bc51-169717ea55ca admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:14.666458 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8ee5b3a-974c-4a24-bc51-169717ea55ca admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:14.667113 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8ee5b3a-974c-4a24-bc51-169717ea55ca admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:14.668740 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8ee5b3a-974c-4a24-bc51-169717ea55ca admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:14.680531 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1359/2730] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 01:37:14 2022] GET /identity/v3/roles => generated 1420 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 28 01:37:14.684895 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e5555bd-f45b-43fa-b461-4ef42b323649 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:14.695578 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e5555bd-f45b-43fa-b461-4ef42b323649 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:14.700803 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cf84bcf-4376-4352-a8af-d122e3e8de9b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:14.701451 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cf84bcf-4376-4352-a8af-d122e3e8de9b admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:14.701729 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cf84bcf-4376-4352-a8af-d122e3e8de9b admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:14.702000 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cf84bcf-4376-4352-a8af-d122e3e8de9b admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:14.702640 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cf84bcf-4376-4352-a8af-d122e3e8de9b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:14.704362 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cf84bcf-4376-4352-a8af-d122e3e8de9b admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:14.718253 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1372/2731] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 01:37:14 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 01:37:14.722155 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8ee5b3a-974c-4a24-bc51-169717ea55ca admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:14.732553 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8ee5b3a-974c-4a24-bc51-169717ea55ca admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:14.738441 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2a16dfb-6ae1-4e34-9588-0e3eab3e0c95 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:14.739122 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2a16dfb-6ae1-4e34-9588-0e3eab3e0c95 admin admin] REQUEST_METHOD: `PUT` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:14.739480 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2a16dfb-6ae1-4e34-9588-0e3eab3e0c95 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:14.739752 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2a16dfb-6ae1-4e34-9588-0e3eab3e0c95 admin admin] PATH_INFO: `/v3/projects/95a408f00f144e509f463adc21829bfd/users/927b321e0aeb4d4b8d24b1dab729855e/roles/0487ef7e943e4701915cb8ce0f2899d7` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:14.760314 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2a16dfb-6ae1-4e34-9588-0e3eab3e0c95 admin admin] RBAC: Authorizing `identity:create_grant(project_id=95a408f00f144e509f463adc21829bfd, user_id=927b321e0aeb4d4b8d24b1dab729855e, role_id=0487ef7e943e4701915cb8ce0f2899d7)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:14.762998 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2a16dfb-6ae1-4e34-9588-0e3eab3e0c95 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:14.777242 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1360/2732] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 01:37:14 2022] PUT /identity/v3/projects/95a408f00f144e509f463adc21829bfd/users/927b321e0aeb4d4b8d24b1dab729855e/roles/0487ef7e943e4701915cb8ce0f2899d7 => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:37:14.787954 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2edbbcfb-f208-4642-8ff9-8a3e925fa835 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:14.788236 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2edbbcfb-f208-4642-8ff9-8a3e925fa835 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:14.788503 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2edbbcfb-f208-4642-8ff9-8a3e925fa835 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:14.818855 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-2edbbcfb-f208-4642-8ff9-8a3e925fa835 None None] MFA Rules not processed for user `927b321e0aeb4d4b8d24b1dab729855e`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:37:14.820192 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-2edbbcfb-f208-4642-8ff9-8a3e925fa835 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:37:14.853995 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=95,keystone:DELETE=2,keystone:INSERT=3 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:37:14.895830 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1373/2733] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:37:14 2022] POST /identity/v3/auth/tokens => generated 3278 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:37:14.900537 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2a16dfb-6ae1-4e34-9588-0e3eab3e0c95 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:14.937996 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:37:14.979963 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2a16dfb-6ae1-4e34-9588-0e3eab3e0c95 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:14.984883 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94cf121b-a7f4-4dbb-aa35-7aeb8f906e79 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:14.985534 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94cf121b-a7f4-4dbb-aa35-7aeb8f906e79 admin admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:14.985832 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94cf121b-a7f4-4dbb-aa35-7aeb8f906e79 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:14.986404 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94cf121b-a7f4-4dbb-aa35-7aeb8f906e79 admin admin] PATH_INFO: `/v3/projects` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:14.986703 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94cf121b-a7f4-4dbb-aa35-7aeb8f906e79 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:14.988617 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94cf121b-a7f4-4dbb-aa35-7aeb8f906e79 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:15.018681 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1361/2734] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 01:37:14 2022] POST /identity/v3/projects => generated 401 bytes in 119 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 01:37:15.022902 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2edbbcfb-f208-4642-8ff9-8a3e925fa835 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:15.035030 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2edbbcfb-f208-4642-8ff9-8a3e925fa835 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:15.097291 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1970a6b6-24b2-49fa-a8a8-f5dca28c619e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:15.097916 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1970a6b6-24b2-49fa-a8a8-f5dca28c619e admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:15.098223 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1970a6b6-24b2-49fa-a8a8-f5dca28c619e admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:15.098497 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1970a6b6-24b2-49fa-a8a8-f5dca28c619e admin admin] PATH_INFO: `/v3/users` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:15.099165 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1970a6b6-24b2-49fa-a8a8-f5dca28c619e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:15.101369 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1970a6b6-24b2-49fa-a8a8-f5dca28c619e admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:15.132796 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1374/2735] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 01:37:15 2022] POST /identity/v3/users => generated 307 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 01:37:15.136891 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94cf121b-a7f4-4dbb-aa35-7aeb8f906e79 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:15.148191 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94cf121b-a7f4-4dbb-aa35-7aeb8f906e79 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:15.153483 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21c32682-a1cd-4bf0-94ca-267d013153cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:15.153667 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c32682-a1cd-4bf0-94ca-267d013153cb admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:15.153942 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c32682-a1cd-4bf0-94ca-267d013153cb admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:15.154207 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c32682-a1cd-4bf0-94ca-267d013153cb admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:15.154874 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21c32682-a1cd-4bf0-94ca-267d013153cb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:15.156481 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21c32682-a1cd-4bf0-94ca-267d013153cb admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:15.168240 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1362/2736] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 01:37:15 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 01:37:15.172451 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1970a6b6-24b2-49fa-a8a8-f5dca28c619e admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:15.186424 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1970a6b6-24b2-49fa-a8a8-f5dca28c619e admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:15.191384 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d7d1ad1-0547-47be-8ce0-944dc6b40398 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:15.192052 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d7d1ad1-0547-47be-8ce0-944dc6b40398 admin admin] REQUEST_METHOD: `PUT` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:15.192332 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d7d1ad1-0547-47be-8ce0-944dc6b40398 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:15.192604 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d7d1ad1-0547-47be-8ce0-944dc6b40398 admin admin] PATH_INFO: `/v3/projects/7fb41cb004f24bbdae2a68e6930dfd4c/users/ee0d56d02eff4e68b2d6c004c3f7aa6e/roles/f08d6043261747e3b4763c744eef7dfb` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:15.207939 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d7d1ad1-0547-47be-8ce0-944dc6b40398 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7fb41cb004f24bbdae2a68e6930dfd4c, user_id=ee0d56d02eff4e68b2d6c004c3f7aa6e, role_id=f08d6043261747e3b4763c744eef7dfb)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:15.210650 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d7d1ad1-0547-47be-8ce0-944dc6b40398 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:15.226110 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1375/2737] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 01:37:15 2022] PUT /identity/v3/projects/7fb41cb004f24bbdae2a68e6930dfd4c/users/ee0d56d02eff4e68b2d6c004c3f7aa6e/roles/f08d6043261747e3b4763c744eef7dfb => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:37:15.230386 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21c32682-a1cd-4bf0-94ca-267d013153cb admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:15.240751 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21c32682-a1cd-4bf0-94ca-267d013153cb admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:15.301377 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd65d7b2-3d10-4514-90c5-25988c05c640 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:15.301983 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd65d7b2-3d10-4514-90c5-25988c05c640 admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:15.302271 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd65d7b2-3d10-4514-90c5-25988c05c640 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:15.302567 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd65d7b2-3d10-4514-90c5-25988c05c640 admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:15.303198 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd65d7b2-3d10-4514-90c5-25988c05c640 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:15.305208 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd65d7b2-3d10-4514-90c5-25988c05c640 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:15.317277 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1363/2738] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 01:37:15 2022] GET /identity/v3/roles => generated 1420 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 01:37:15.322554 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d7d1ad1-0547-47be-8ce0-944dc6b40398 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:15.333141 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d7d1ad1-0547-47be-8ce0-944dc6b40398 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:15.338700 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b61c554-d8f2-4172-98c3-a0de1e0135f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:15.339402 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b61c554-d8f2-4172-98c3-a0de1e0135f5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:15.339680 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b61c554-d8f2-4172-98c3-a0de1e0135f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:15.339952 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b61c554-d8f2-4172-98c3-a0de1e0135f5 admin admin] PATH_INFO: `/v3/projects/7fb41cb004f24bbdae2a68e6930dfd4c/users/ee0d56d02eff4e68b2d6c004c3f7aa6e/roles/f08d6043261747e3b4763c744eef7dfb` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:15.342268 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b61c554-d8f2-4172-98c3-a0de1e0135f5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7fb41cb004f24bbdae2a68e6930dfd4c, user_id=ee0d56d02eff4e68b2d6c004c3f7aa6e, role_id=f08d6043261747e3b4763c744eef7dfb)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:15.345704 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b61c554-d8f2-4172-98c3-a0de1e0135f5 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:15.359903 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1376/2739] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 01:37:15 2022] PUT /identity/v3/projects/7fb41cb004f24bbdae2a68e6930dfd4c/users/ee0d56d02eff4e68b2d6c004c3f7aa6e/roles/f08d6043261747e3b4763c744eef7dfb => generated 0 bytes in 38 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:37:15.372571 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907a23dc-36e0-4635-9ef4-badb5898f934 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:15.373183 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907a23dc-36e0-4635-9ef4-badb5898f934 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:15.373395 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907a23dc-36e0-4635-9ef4-badb5898f934 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:15.405617 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-907a23dc-36e0-4635-9ef4-badb5898f934 None None] MFA Rules not processed for user `ee0d56d02eff4e68b2d6c004c3f7aa6e`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:37:15.406760 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-907a23dc-36e0-4635-9ef4-badb5898f934 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:37:15.482274 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1364/2740] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:37:15 2022] POST /identity/v3/auth/tokens => generated 3345 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:37:15.493487 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b61c554-d8f2-4172-98c3-a0de1e0135f5 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:15.495392 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-5b61c554-d8f2-4172-98c3-a0de1e0135f5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:37:15.558848 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b61c554-d8f2-4172-98c3-a0de1e0135f5 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:15.562715 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa601e1e-22c7-483c-9f53-8f2e6b83f00a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:15.563329 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa601e1e-22c7-483c-9f53-8f2e6b83f00a service cinder] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:15.563605 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa601e1e-22c7-483c-9f53-8f2e6b83f00a service cinder] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:15.563876 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa601e1e-22c7-483c-9f53-8f2e6b83f00a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:15.573705 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa601e1e-22c7-483c-9f53-8f2e6b83f00a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:15.575159 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa601e1e-22c7-483c-9f53-8f2e6b83f00a service cinder] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:15.585622 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1377/2741] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:37:15 2022] GET /identity/v3/auth/tokens => generated 3345 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:37:15.630465 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-907a23dc-36e0-4635-9ef4-badb5898f934 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:15.632345 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-907a23dc-36e0-4635-9ef4-badb5898f934 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:37:15.667391 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:37:15.678973 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa601e1e-22c7-483c-9f53-8f2e6b83f00a service cinder] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:15.689003 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa601e1e-22c7-483c-9f53-8f2e6b83f00a service cinder] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:15.697573 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d7efb55-337e-4c70-b409-5ae74f56facc service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:15.698180 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d7efb55-337e-4c70-b409-5ae74f56facc service cinder] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:15.698463 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d7efb55-337e-4c70-b409-5ae74f56facc service cinder] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:15.698734 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d7efb55-337e-4c70-b409-5ae74f56facc service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:15.707092 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-907a23dc-36e0-4635-9ef4-badb5898f934 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:15.710696 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2546edd2-1273-46a9-9ece-86149a498bef service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:15.711305 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2546edd2-1273-46a9-9ece-86149a498bef service neutron] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:15.711577 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2546edd2-1273-46a9-9ece-86149a498bef service neutron] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:15.711900 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2546edd2-1273-46a9-9ece-86149a498bef service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:15.713672 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-2546edd2-1273-46a9-9ece-86149a498bef service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:37:15.720786 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:37:15.763771 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d7efb55-337e-4c70-b409-5ae74f56facc service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:15.765307 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d7efb55-337e-4c70-b409-5ae74f56facc service cinder] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:15.774641 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1378/2742] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:37:15 2022] GET /identity/v3/auth/tokens => generated 3278 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 28 01:37:15.777823 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2546edd2-1273-46a9-9ece-86149a498bef service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:15.780388 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2546edd2-1273-46a9-9ece-86149a498bef service neutron] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:15.791853 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1365/2743] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:37:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 162 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:37:19.271184 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d7efb55-337e-4c70-b409-5ae74f56facc service cinder] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:19.273507 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-6d7efb55-337e-4c70-b409-5ae74f56facc service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:37:19.291825 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d7efb55-337e-4c70-b409-5ae74f56facc service cinder] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:19.300216 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2c18831-fc71-4626-9b91-e77370ad3038 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:19.301082 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2c18831-fc71-4626-9b91-e77370ad3038 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:19.301360 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2c18831-fc71-4626-9b91-e77370ad3038 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:19.301628 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2c18831-fc71-4626-9b91-e77370ad3038 admin admin] PATH_INFO: `/v3/users/1ab2fcb3759045c7a076e5c8222a421d` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:19.302761 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2c18831-fc71-4626-9b91-e77370ad3038 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1ab2fcb3759045c7a076e5c8222a421d)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:19.305336 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2c18831-fc71-4626-9b91-e77370ad3038 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:19.358475 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-d2c18831-fc71-4626-9b91-e77370ad3038 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1ab2fcb3759045c7a076e5c8222a421d'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:19.359145 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-d2c18831-fc71-4626-9b91-e77370ad3038 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1ab2fcb3759045c7a076e5c8222a421d'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:19.370190 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-d2c18831-fc71-4626-9b91-e77370ad3038 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1ab2fcb3759045c7a076e5c8222a421d'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:19.377669 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-d2c18831-fc71-4626-9b91-e77370ad3038 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1ab2fcb3759045c7a076e5c8222a421d'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:19.378383 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-d2c18831-fc71-4626-9b91-e77370ad3038 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1ab2fcb3759045c7a076e5c8222a421d'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:19.396353 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1379/2744] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 01:37:19 2022] DELETE /identity/v3/users/1ab2fcb3759045c7a076e5c8222a421d => generated 0 bytes in 126 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:37:19.577905 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2546edd2-1273-46a9-9ece-86149a498bef service neutron] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:19.588832 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2546edd2-1273-46a9-9ece-86149a498bef service neutron] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:19.654602 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-153040af-a201-44c7-aeb1-3447a9e3d360 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:19.655345 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-153040af-a201-44c7-aeb1-3447a9e3d360 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:19.655706 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-153040af-a201-44c7-aeb1-3447a9e3d360 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:19.656037 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-153040af-a201-44c7-aeb1-3447a9e3d360 admin admin] PATH_INFO: `/v3/projects/6f3acee4af62483ab1f2cbf3a47b09e1` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:19.657338 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-153040af-a201-44c7-aeb1-3447a9e3d360 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6f3acee4af62483ab1f2cbf3a47b09e1)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:19.659318 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-153040af-a201-44c7-aeb1-3447a9e3d360 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:19.708978 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-153040af-a201-44c7-aeb1-3447a9e3d360 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6f3acee4af62483ab1f2cbf3a47b09e1', 'request_id': 'req-153040af-a201-44c7-aeb1-3447a9e3d360'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:19.714653 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-153040af-a201-44c7-aeb1-3447a9e3d360 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6f3acee4af62483ab1f2cbf3a47b09e1', 'request_id': 'req-153040af-a201-44c7-aeb1-3447a9e3d360'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:19.722364 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-153040af-a201-44c7-aeb1-3447a9e3d360 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6f3acee4af62483ab1f2cbf3a47b09e1', 'request_id': 'req-153040af-a201-44c7-aeb1-3447a9e3d360'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:19.751938 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-153040af-a201-44c7-aeb1-3447a9e3d360 admin admin] The token cache is being invalidate because project 6f3acee4af62483ab1f2cbf3a47b09e1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81507) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 01:37:19.752220 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-153040af-a201-44c7-aeb1-3447a9e3d360 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:19.752925 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-153040af-a201-44c7-aeb1-3447a9e3d360 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:19.754713 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1366/2745] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 01:37:19 2022] DELETE /identity/v3/projects/6f3acee4af62483ab1f2cbf3a47b09e1 => generated 0 bytes in 177 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:37:19.761247 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9f5b1d3-08bf-49f9-b5b3-0b1f5bca968c None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:19.761659 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9f5b1d3-08bf-49f9-b5b3-0b1f5bca968c None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:19.762072 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9f5b1d3-08bf-49f9-b5b3-0b1f5bca968c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:19.782240 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-b9f5b1d3-08bf-49f9-b5b3-0b1f5bca968c None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:37:19.783348 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-b9f5b1d3-08bf-49f9-b5b3-0b1f5bca968c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:37:19.861694 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1380/2746] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:37:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:37:19.870508 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e8c1204-7068-4cb4-9a86-0956d0426826 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:19.870791 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e8c1204-7068-4cb4-9a86-0956d0426826 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:19.871303 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e8c1204-7068-4cb4-9a86-0956d0426826 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:19.882973 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:37:19.905954 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-2e8c1204-7068-4cb4-9a86-0956d0426826 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:37:19.908067 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-2e8c1204-7068-4cb4-9a86-0956d0426826 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:37:19.923930 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1367/2747] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:37:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 51 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:37:19.925366 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9f5b1d3-08bf-49f9-b5b3-0b1f5bca968c None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:19.937330 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9f5b1d3-08bf-49f9-b5b3-0b1f5bca968c None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:19.942660 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e293f878-cc58-4ce2-a691-3c1b6e86f9af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:19.943543 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e293f878-cc58-4ce2-a691-3c1b6e86f9af admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:19.944030 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e293f878-cc58-4ce2-a691-3c1b6e86f9af admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:19.944343 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e293f878-cc58-4ce2-a691-3c1b6e86f9af admin admin] PATH_INFO: `/v3/domains` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:19.945323 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e293f878-cc58-4ce2-a691-3c1b6e86f9af admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 01:37:19.945718 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e293f878-cc58-4ce2-a691-3c1b6e86f9af admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:19.947774 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e293f878-cc58-4ce2-a691-3c1b6e86f9af admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:19.966370 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1381/2748] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Aug 28 01:37:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 28 01:37:19.967778 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e8c1204-7068-4cb4-9a86-0956d0426826 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:19.978843 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e8c1204-7068-4cb4-9a86-0956d0426826 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:19.987231 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4f80abf-e1e7-49f7-a4cb-5407c630628a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:19.987987 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4f80abf-e1e7-49f7-a4cb-5407c630628a admin admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:19.988274 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4f80abf-e1e7-49f7-a4cb-5407c630628a admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:19.988757 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4f80abf-e1e7-49f7-a4cb-5407c630628a admin admin] PATH_INFO: `/v3/projects` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:19.989376 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4f80abf-e1e7-49f7-a4cb-5407c630628a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:19.991298 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4f80abf-e1e7-49f7-a4cb-5407c630628a admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:20.024112 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1368/2749] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 01:37:19 2022] POST /identity/v3/projects => generated 403 bytes in 57 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 01:37:20.028013 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e293f878-cc58-4ce2-a691-3c1b6e86f9af admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:20.039566 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e293f878-cc58-4ce2-a691-3c1b6e86f9af admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:20.040400 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4f80abf-e1e7-49f7-a4cb-5407c630628a admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:20.042513 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-b4f80abf-e1e7-49f7-a4cb-5407c630628a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:37:20.068400 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:37:20.124448 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acd2660b-ded4-4968-92da-22ba6b883f76 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:20.125097 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acd2660b-ded4-4968-92da-22ba6b883f76 admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:20.128135 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acd2660b-ded4-4968-92da-22ba6b883f76 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:20.128135 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acd2660b-ded4-4968-92da-22ba6b883f76 admin admin] PATH_INFO: `/v3/users` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:20.128761 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acd2660b-ded4-4968-92da-22ba6b883f76 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:20.131084 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acd2660b-ded4-4968-92da-22ba6b883f76 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:20.139937 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4f80abf-e1e7-49f7-a4cb-5407c630628a admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:20.146248 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c26e97db-ad8d-4e04-97b8-4670890d3301 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:20.146948 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c26e97db-ad8d-4e04-97b8-4670890d3301 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:20.147218 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c26e97db-ad8d-4e04-97b8-4670890d3301 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:20.147493 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c26e97db-ad8d-4e04-97b8-4670890d3301 admin admin] PATH_INFO: `/v3/users/927b321e0aeb4d4b8d24b1dab729855e` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:20.148946 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c26e97db-ad8d-4e04-97b8-4670890d3301 admin admin] RBAC: Authorizing `identity:delete_user(user_id=927b321e0aeb4d4b8d24b1dab729855e)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:20.151002 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c26e97db-ad8d-4e04-97b8-4670890d3301 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:20.158489 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1382/2750] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 01:37:20 2022] POST /identity/v3/users => generated 302 bytes in 131 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 01:37:20.162331 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acd2660b-ded4-4968-92da-22ba6b883f76 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:20.173746 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acd2660b-ded4-4968-92da-22ba6b883f76 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:20.181774 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eab9d1a-68d8-4c91-807c-c725d0b354ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:20.182521 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eab9d1a-68d8-4c91-807c-c725d0b354ed admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:20.182879 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eab9d1a-68d8-4c91-807c-c725d0b354ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:20.183152 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eab9d1a-68d8-4c91-807c-c725d0b354ed admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:20.183837 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eab9d1a-68d8-4c91-807c-c725d0b354ed admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:20.185579 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eab9d1a-68d8-4c91-807c-c725d0b354ed admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:20.199193 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1383/2751] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 01:37:20 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 01:37:20.203285 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eab9d1a-68d8-4c91-807c-c725d0b354ed admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:20.213385 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-c26e97db-ad8d-4e04-97b8-4670890d3301 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '927b321e0aeb4d4b8d24b1dab729855e'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:20.214426 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eab9d1a-68d8-4c91-807c-c725d0b354ed admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:20.227985 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-c26e97db-ad8d-4e04-97b8-4670890d3301 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '927b321e0aeb4d4b8d24b1dab729855e'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:20.228846 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1731d14-08b6-473f-8462-f83ad439f4c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:20.229860 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1731d14-08b6-473f-8462-f83ad439f4c1 admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:20.230336 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1731d14-08b6-473f-8462-f83ad439f4c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:20.231346 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1731d14-08b6-473f-8462-f83ad439f4c1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:20.232593 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1731d14-08b6-473f-8462-f83ad439f4c1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:20.234751 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1731d14-08b6-473f-8462-f83ad439f4c1 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:20.238470 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-c26e97db-ad8d-4e04-97b8-4670890d3301 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '927b321e0aeb4d4b8d24b1dab729855e'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:20.249401 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-c26e97db-ad8d-4e04-97b8-4670890d3301 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '927b321e0aeb4d4b8d24b1dab729855e'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:20.249647 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1384/2752] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 01:37:20 2022] GET /identity/v3/roles => generated 1420 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 01:37:20.250549 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-c26e97db-ad8d-4e04-97b8-4670890d3301 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '927b321e0aeb4d4b8d24b1dab729855e'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:20.254196 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1731d14-08b6-473f-8462-f83ad439f4c1 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:20.257616 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1369/2753] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 01:37:20 2022] DELETE /identity/v3/users/927b321e0aeb4d4b8d24b1dab729855e => generated 0 bytes in 218 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 28 01:37:20.258658 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-f1731d14-08b6-473f-8462-f83ad439f4c1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:37:20.262391 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1b16bf0-3204-4931-821c-f9fa5e079d4f None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:20.262679 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1b16bf0-3204-4931-821c-f9fa5e079d4f None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:20.262955 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1b16bf0-3204-4931-821c-f9fa5e079d4f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:20.278338 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1731d14-08b6-473f-8462-f83ad439f4c1 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:20.296259 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-d1b16bf0-3204-4931-821c-f9fa5e079d4f None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:37:20.297465 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-d1b16bf0-3204-4931-821c-f9fa5e079d4f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:37:20.343380 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc6aa6c9-ae2b-4aa9-afc2-45bf346f1e58 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:20.344073 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc6aa6c9-ae2b-4aa9-afc2-45bf346f1e58 admin admin] REQUEST_METHOD: `PUT` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:20.344396 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc6aa6c9-ae2b-4aa9-afc2-45bf346f1e58 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:20.344694 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc6aa6c9-ae2b-4aa9-afc2-45bf346f1e58 admin admin] PATH_INFO: `/v3/projects/315c865ba0e744059bca147998fd1302/users/52a472bb99494401ac5d099b4cfca133/roles/0487ef7e943e4701915cb8ce0f2899d7` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:20.355077 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:37:20.360110 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc6aa6c9-ae2b-4aa9-afc2-45bf346f1e58 admin admin] RBAC: Authorizing `identity:create_grant(project_id=315c865ba0e744059bca147998fd1302, user_id=52a472bb99494401ac5d099b4cfca133, role_id=0487ef7e943e4701915cb8ce0f2899d7)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:20.363180 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc6aa6c9-ae2b-4aa9-afc2-45bf346f1e58 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:20.370548 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1370/2754] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:37:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:37:20.380860 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1b16bf0-3204-4931-821c-f9fa5e079d4f None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:20.381439 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1385/2755] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 01:37:20 2022] PUT /identity/v3/projects/315c865ba0e744059bca147998fd1302/users/52a472bb99494401ac5d099b4cfca133/roles/0487ef7e943e4701915cb8ce0f2899d7 => generated 0 bytes in 128 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:37:20.382460 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-d1b16bf0-3204-4931-821c-f9fa5e079d4f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:37:20.386518 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09b23df7-b084-48ec-8cf3-2d6e0e582235 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:20.386805 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09b23df7-b084-48ec-8cf3-2d6e0e582235 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:20.387078 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09b23df7-b084-48ec-8cf3-2d6e0e582235 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:20.406632 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-09b23df7-b084-48ec-8cf3-2d6e0e582235 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:37:20.407890 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-09b23df7-b084-48ec-8cf3-2d6e0e582235 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:37:20.440392 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1b16bf0-3204-4931-821c-f9fa5e079d4f None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:20.473007 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1386/2756] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:37:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 28 01:37:20.484840 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09b23df7-b084-48ec-8cf3-2d6e0e582235 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:20.495580 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41bc271b-9836-4870-a6a4-b04a8ad05883 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:20.496267 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41bc271b-9836-4870-a6a4-b04a8ad05883 service neutron] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:20.496675 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41bc271b-9836-4870-a6a4-b04a8ad05883 service neutron] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:20.496936 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41bc271b-9836-4870-a6a4-b04a8ad05883 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:20.502381 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:37:20.508219 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41bc271b-9836-4870-a6a4-b04a8ad05883 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:20.509903 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41bc271b-9836-4870-a6a4-b04a8ad05883 service neutron] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:20.520115 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1371/2757] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:37:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 140 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:37:20.548354 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09b23df7-b084-48ec-8cf3-2d6e0e582235 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:20.552072 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bb5e63d-1b18-43f2-bf4c-c86482905909 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:20.552756 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb5e63d-1b18-43f2-bf4c-c86482905909 service neutron] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:20.553144 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb5e63d-1b18-43f2-bf4c-c86482905909 service neutron] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:20.553420 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb5e63d-1b18-43f2-bf4c-c86482905909 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:20.563895 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bb5e63d-1b18-43f2-bf4c-c86482905909 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:20.565400 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bb5e63d-1b18-43f2-bf4c-c86482905909 service neutron] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:20.576498 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1387/2758] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:37:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 28 01:37:20.873965 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41bc271b-9836-4870-a6a4-b04a8ad05883 service neutron] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:20.876253 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-41bc271b-9836-4870-a6a4-b04a8ad05883 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:37:20.892981 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41bc271b-9836-4870-a6a4-b04a8ad05883 service neutron] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:20.898113 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbe14c89-2462-452a-bc03-98fcc9e3e804 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:20.898851 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe14c89-2462-452a-bc03-98fcc9e3e804 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:20.899140 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe14c89-2462-452a-bc03-98fcc9e3e804 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:20.899410 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe14c89-2462-452a-bc03-98fcc9e3e804 admin admin] PATH_INFO: `/v3/projects/95a408f00f144e509f463adc21829bfd` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:20.900753 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbe14c89-2462-452a-bc03-98fcc9e3e804 admin admin] RBAC: Authorizing `identity:delete_project(project_id=95a408f00f144e509f463adc21829bfd)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:20.902759 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbe14c89-2462-452a-bc03-98fcc9e3e804 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:20.958711 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-dbe14c89-2462-452a-bc03-98fcc9e3e804 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '95a408f00f144e509f463adc21829bfd', 'request_id': 'req-dbe14c89-2462-452a-bc03-98fcc9e3e804'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:20.970333 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-dbe14c89-2462-452a-bc03-98fcc9e3e804 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '95a408f00f144e509f463adc21829bfd', 'request_id': 'req-dbe14c89-2462-452a-bc03-98fcc9e3e804'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:20.976812 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-dbe14c89-2462-452a-bc03-98fcc9e3e804 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '95a408f00f144e509f463adc21829bfd', 'request_id': 'req-dbe14c89-2462-452a-bc03-98fcc9e3e804'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:20.992389 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-dbe14c89-2462-452a-bc03-98fcc9e3e804 admin admin] The token cache is being invalidate because project 95a408f00f144e509f463adc21829bfd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81507) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 01:37:20.992683 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-dbe14c89-2462-452a-bc03-98fcc9e3e804 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:20.993355 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-dbe14c89-2462-452a-bc03-98fcc9e3e804 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:20.994855 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1372/2759] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 01:37:20 2022] DELETE /identity/v3/projects/95a408f00f144e509f463adc21829bfd => generated 0 bytes in 122 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:37:20.999093 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bb5e63d-1b18-43f2-bf4c-c86482905909 service neutron] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:21.009835 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bb5e63d-1b18-43f2-bf4c-c86482905909 service neutron] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:21.079614 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10fb8f93-a762-40d7-a5ac-4ac1746b72ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:21.080270 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10fb8f93-a762-40d7-a5ac-4ac1746b72ac admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:21.080584 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10fb8f93-a762-40d7-a5ac-4ac1746b72ac admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:21.080855 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10fb8f93-a762-40d7-a5ac-4ac1746b72ac admin admin] PATH_INFO: `/v3/users/ee0d56d02eff4e68b2d6c004c3f7aa6e` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:21.082290 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10fb8f93-a762-40d7-a5ac-4ac1746b72ac admin admin] RBAC: Authorizing `identity:delete_user(user_id=ee0d56d02eff4e68b2d6c004c3f7aa6e)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:21.084251 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10fb8f93-a762-40d7-a5ac-4ac1746b72ac admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:21.148016 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-10fb8f93-a762-40d7-a5ac-4ac1746b72ac admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ee0d56d02eff4e68b2d6c004c3f7aa6e'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:21.149038 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-10fb8f93-a762-40d7-a5ac-4ac1746b72ac admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ee0d56d02eff4e68b2d6c004c3f7aa6e'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:21.160308 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-10fb8f93-a762-40d7-a5ac-4ac1746b72ac admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ee0d56d02eff4e68b2d6c004c3f7aa6e'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:21.168391 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-10fb8f93-a762-40d7-a5ac-4ac1746b72ac admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ee0d56d02eff4e68b2d6c004c3f7aa6e'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:21.169119 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-10fb8f93-a762-40d7-a5ac-4ac1746b72ac admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ee0d56d02eff4e68b2d6c004c3f7aa6e'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:21.188906 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1388/2760] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 01:37:20 2022] DELETE /identity/v3/users/ee0d56d02eff4e68b2d6c004c3f7aa6e => generated 0 bytes in 190 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:37:21.340297 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-201a20a5-6e67-43ff-9727-aa370989b566 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:21.340581 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-201a20a5-6e67-43ff-9727-aa370989b566 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:21.340853 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-201a20a5-6e67-43ff-9727-aa370989b566 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:21.385752 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-201a20a5-6e67-43ff-9727-aa370989b566 None None] MFA Rules not processed for user `52a472bb99494401ac5d099b4cfca133`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:37:21.387596 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-201a20a5-6e67-43ff-9727-aa370989b566 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:37:21.427537 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:37:21.496216 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1373/2761] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:37:21 2022] POST /identity/v3/auth/tokens => generated 3280 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:37:21.497718 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10fb8f93-a762-40d7-a5ac-4ac1746b72ac admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:21.499309 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-10fb8f93-a762-40d7-a5ac-4ac1746b72ac admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:37:21.547798 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:37:21.590440 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-201a20a5-6e67-43ff-9727-aa370989b566 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:21.591944 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-201a20a5-6e67-43ff-9727-aa370989b566 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:37:21.605650 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10fb8f93-a762-40d7-a5ac-4ac1746b72ac admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:21.612953 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6bd08b5-c26e-4bcf-964e-ff1376e9ddc5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:21.613612 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6bd08b5-c26e-4bcf-964e-ff1376e9ddc5 admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:21.613943 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6bd08b5-c26e-4bcf-964e-ff1376e9ddc5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:21.614145 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6bd08b5-c26e-4bcf-964e-ff1376e9ddc5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:21.614785 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6bd08b5-c26e-4bcf-964e-ff1376e9ddc5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:21.617163 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6bd08b5-c26e-4bcf-964e-ff1376e9ddc5 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:21.621515 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-201a20a5-6e67-43ff-9727-aa370989b566 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:21.626848 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e200c974-bcda-4a62-8f72-b95c3dbd8825 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:21.627974 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e200c974-bcda-4a62-8f72-b95c3dbd8825 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:21.628262 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e200c974-bcda-4a62-8f72-b95c3dbd8825 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:21.628539 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e200c974-bcda-4a62-8f72-b95c3dbd8825 admin admin] PATH_INFO: `/v3/projects/7fb41cb004f24bbdae2a68e6930dfd4c` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:21.629890 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e200c974-bcda-4a62-8f72-b95c3dbd8825 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7fb41cb004f24bbdae2a68e6930dfd4c)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:21.631810 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e200c974-bcda-4a62-8f72-b95c3dbd8825 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:21.652817 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1389/2762] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 01:37:21 2022] POST /identity/v3/projects => generated 405 bytes in 155 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 01:37:21.656484 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6bd08b5-c26e-4bcf-964e-ff1376e9ddc5 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:21.694604 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-e200c974-bcda-4a62-8f72-b95c3dbd8825 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7fb41cb004f24bbdae2a68e6930dfd4c', 'request_id': 'req-e200c974-bcda-4a62-8f72-b95c3dbd8825'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:21.700090 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-e200c974-bcda-4a62-8f72-b95c3dbd8825 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7fb41cb004f24bbdae2a68e6930dfd4c', 'request_id': 'req-e200c974-bcda-4a62-8f72-b95c3dbd8825'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:21.706497 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-e200c974-bcda-4a62-8f72-b95c3dbd8825 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7fb41cb004f24bbdae2a68e6930dfd4c', 'request_id': 'req-e200c974-bcda-4a62-8f72-b95c3dbd8825'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:21.721481 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-e200c974-bcda-4a62-8f72-b95c3dbd8825 admin admin] The token cache is being invalidate because project 7fb41cb004f24bbdae2a68e6930dfd4c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81507) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 01:37:21.721771 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-e200c974-bcda-4a62-8f72-b95c3dbd8825 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:21.722362 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6bd08b5-c26e-4bcf-964e-ff1376e9ddc5 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:21.722495 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-e200c974-bcda-4a62-8f72-b95c3dbd8825 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:21.725664 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1374/2763] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 01:37:21 2022] DELETE /identity/v3/projects/7fb41cb004f24bbdae2a68e6930dfd4c => generated 0 bytes in 136 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:37:21.729844 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c94d522-7fad-4bb5-9599-53d1a65b906f None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:21.730197 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c94d522-7fad-4bb5-9599-53d1a65b906f None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:21.730538 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c94d522-7fad-4bb5-9599-53d1a65b906f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:21.757968 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-1c94d522-7fad-4bb5-9599-53d1a65b906f None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:37:21.759238 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-1c94d522-7fad-4bb5-9599-53d1a65b906f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:37:21.794912 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e49c7960-f99a-458f-b9c5-5c065ad99e41 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:21.795532 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e49c7960-f99a-458f-b9c5-5c065ad99e41 admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:21.795841 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e49c7960-f99a-458f-b9c5-5c065ad99e41 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:21.796105 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e49c7960-f99a-458f-b9c5-5c065ad99e41 admin admin] PATH_INFO: `/v3/users` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:21.796751 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e49c7960-f99a-458f-b9c5-5c065ad99e41 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:21.798914 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e49c7960-f99a-458f-b9c5-5c065ad99e41 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:21.827529 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1390/2764] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 01:37:21 2022] POST /identity/v3/users => generated 309 bytes in 169 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 28 01:37:21.829070 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e49c7960-f99a-458f-b9c5-5c065ad99e41 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:21.831050 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-e49c7960-f99a-458f-b9c5-5c065ad99e41 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:37:21.834766 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1375/2765] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:37:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 28 01:37:21.842803 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1cd0836-a0f1-4cfd-bae6-e43d2d600752 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:21.843156 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1cd0836-a0f1-4cfd-bae6-e43d2d600752 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:21.843493 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1cd0836-a0f1-4cfd-bae6-e43d2d600752 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:21.844627 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e49c7960-f99a-458f-b9c5-5c065ad99e41 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:21.849908 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d7249fe-0dd9-4bb5-bbcb-f24fea69a71e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:21.850670 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d7249fe-0dd9-4bb5-bbcb-f24fea69a71e admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:21.851117 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d7249fe-0dd9-4bb5-bbcb-f24fea69a71e admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:21.851387 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d7249fe-0dd9-4bb5-bbcb-f24fea69a71e admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:21.852120 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d7249fe-0dd9-4bb5-bbcb-f24fea69a71e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:21.853854 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d7249fe-0dd9-4bb5-bbcb-f24fea69a71e admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:21.865510 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-d1cd0836-a0f1-4cfd-bae6-e43d2d600752 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:37:21.865915 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1391/2766] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 01:37:21 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 28 01:37:21.869370 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-d1cd0836-a0f1-4cfd-bae6-e43d2d600752 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:37:21.870835 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d7249fe-0dd9-4bb5-bbcb-f24fea69a71e admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:21.879636 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1376/2767] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:37:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:37:21.883640 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1cd0836-a0f1-4cfd-bae6-e43d2d600752 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:21.883781 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d7249fe-0dd9-4bb5-bbcb-f24fea69a71e admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:21.889076 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e387b5c-24bb-4177-8d35-6e78e488f458 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:21.889749 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e387b5c-24bb-4177-8d35-6e78e488f458 admin admin] REQUEST_METHOD: `PUT` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:21.890253 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e387b5c-24bb-4177-8d35-6e78e488f458 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:21.890353 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e387b5c-24bb-4177-8d35-6e78e488f458 admin admin] PATH_INFO: `/v3/projects/239722b27796406ca1638dcdeb05d6ee/users/5317b0b17103487daa2b71273c4ec2c0/roles/f08d6043261747e3b4763c744eef7dfb` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:21.894264 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1cd0836-a0f1-4cfd-bae6-e43d2d600752 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:21.899831 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f27d0af-e34e-48fd-9f8f-9582034d1d1f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:21.900563 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f27d0af-e34e-48fd-9f8f-9582034d1d1f admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:21.901108 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f27d0af-e34e-48fd-9f8f-9582034d1d1f admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:21.901445 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f27d0af-e34e-48fd-9f8f-9582034d1d1f admin admin] PATH_INFO: `/v3/domains` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:21.902267 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f27d0af-e34e-48fd-9f8f-9582034d1d1f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 01:37:21.902705 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f27d0af-e34e-48fd-9f8f-9582034d1d1f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:21.904498 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f27d0af-e34e-48fd-9f8f-9582034d1d1f admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:21.908543 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e387b5c-24bb-4177-8d35-6e78e488f458 admin admin] RBAC: Authorizing `identity:create_grant(project_id=239722b27796406ca1638dcdeb05d6ee, user_id=5317b0b17103487daa2b71273c4ec2c0, role_id=f08d6043261747e3b4763c744eef7dfb)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:21.911803 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e387b5c-24bb-4177-8d35-6e78e488f458 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:21.918337 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1377/2768] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Aug 28 01:37:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Aug 28 01:37:21.923082 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f27d0af-e34e-48fd-9f8f-9582034d1d1f admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:21.934849 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1392/2769] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 01:37:21 2022] PUT /identity/v3/projects/239722b27796406ca1638dcdeb05d6ee/users/5317b0b17103487daa2b71273c4ec2c0/roles/f08d6043261747e3b4763c744eef7dfb => generated 0 bytes in 65 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:37:21.935075 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f27d0af-e34e-48fd-9f8f-9582034d1d1f admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:21.939198 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e387b5c-24bb-4177-8d35-6e78e488f458 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:21.969967 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:37:21.973324 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:37:22.007201 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ba1335f-729c-4d2d-b7d5-18e2867db05b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:22.007901 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba1335f-729c-4d2d-b7d5-18e2867db05b admin admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:22.008250 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba1335f-729c-4d2d-b7d5-18e2867db05b admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:22.008605 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba1335f-729c-4d2d-b7d5-18e2867db05b admin admin] PATH_INFO: `/v3/projects` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:22.009329 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ba1335f-729c-4d2d-b7d5-18e2867db05b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:22.011322 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ba1335f-729c-4d2d-b7d5-18e2867db05b admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:22.017583 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e387b5c-24bb-4177-8d35-6e78e488f458 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:22.022988 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55857d11-102f-4d00-8710-7751d313e4cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:22.023686 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55857d11-102f-4d00-8710-7751d313e4cc admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:22.024027 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55857d11-102f-4d00-8710-7751d313e4cc admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:22.024359 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55857d11-102f-4d00-8710-7751d313e4cc admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:22.025175 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55857d11-102f-4d00-8710-7751d313e4cc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:22.027055 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55857d11-102f-4d00-8710-7751d313e4cc admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:22.040999 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1393/2770] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 01:37:21 2022] GET /identity/v3/roles => generated 1420 bytes in 100 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 28 01:37:22.042950 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1378/2771] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 01:37:21 2022] POST /identity/v3/projects => generated 375 bytes in 120 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 01:37:22.043077 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55857d11-102f-4d00-8710-7751d313e4cc admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:22.046852 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ba1335f-729c-4d2d-b7d5-18e2867db05b admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:22.053221 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55857d11-102f-4d00-8710-7751d313e4cc admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:22.116823 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ba1335f-729c-4d2d-b7d5-18e2867db05b admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:22.121865 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ffd007b-c843-4727-90a0-80944f81dbd3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:22.122570 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ffd007b-c843-4727-90a0-80944f81dbd3 admin admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:22.122659 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ffd007b-c843-4727-90a0-80944f81dbd3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:22.122922 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ffd007b-c843-4727-90a0-80944f81dbd3 admin admin] PATH_INFO: `/v3/users` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:22.123311 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44335753-0301-489c-9c32-9d35f74ff24c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:22.123564 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ffd007b-c843-4727-90a0-80944f81dbd3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:22.123990 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44335753-0301-489c-9c32-9d35f74ff24c admin admin] REQUEST_METHOD: `PUT` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:22.124265 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44335753-0301-489c-9c32-9d35f74ff24c admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:22.124536 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44335753-0301-489c-9c32-9d35f74ff24c admin admin] PATH_INFO: `/v3/projects/239722b27796406ca1638dcdeb05d6ee/users/5317b0b17103487daa2b71273c4ec2c0/roles/f08d6043261747e3b4763c744eef7dfb` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:22.125602 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ffd007b-c843-4727-90a0-80944f81dbd3 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:22.127388 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44335753-0301-489c-9c32-9d35f74ff24c admin admin] RBAC: Authorizing `identity:create_grant(project_id=239722b27796406ca1638dcdeb05d6ee, user_id=5317b0b17103487daa2b71273c4ec2c0, role_id=f08d6043261747e3b4763c744eef7dfb)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:22.130227 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44335753-0301-489c-9c32-9d35f74ff24c admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:22.144293 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1394/2772] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 01:37:22 2022] PUT /identity/v3/projects/239722b27796406ca1638dcdeb05d6ee/users/5317b0b17103487daa2b71273c4ec2c0/roles/f08d6043261747e3b4763c744eef7dfb => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 28 01:37:22.154698 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1379/2773] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 01:37:22 2022] POST /identity/v3/users => generated 288 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 28 01:37:22.158786 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44335753-0301-489c-9c32-9d35f74ff24c admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:22.221830 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44335753-0301-489c-9c32-9d35f74ff24c admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:22.226422 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9418ee4d-cce3-4e9f-8834-d79e9cb77047 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:22.227029 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9418ee4d-cce3-4e9f-8834-d79e9cb77047 admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:22.227340 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9418ee4d-cce3-4e9f-8834-d79e9cb77047 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:22.227599 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9418ee4d-cce3-4e9f-8834-d79e9cb77047 admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:22.228208 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9418ee4d-cce3-4e9f-8834-d79e9cb77047 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:22.229978 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9418ee4d-cce3-4e9f-8834-d79e9cb77047 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:22.240360 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1395/2774] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 01:37:22 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 01:37:22.245843 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ffd007b-c843-4727-90a0-80944f81dbd3 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:22.257701 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ffd007b-c843-4727-90a0-80944f81dbd3 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:22.262734 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b47f2315-52c7-4a97-913a-37b70fcb2a53 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:22.263512 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47f2315-52c7-4a97-913a-37b70fcb2a53 admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:22.263786 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47f2315-52c7-4a97-913a-37b70fcb2a53 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:22.264059 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47f2315-52c7-4a97-913a-37b70fcb2a53 admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:22.264725 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b47f2315-52c7-4a97-913a-37b70fcb2a53 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:22.266401 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b47f2315-52c7-4a97-913a-37b70fcb2a53 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:22.277525 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1380/2775] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 01:37:22 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 01:37:22.282135 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9418ee4d-cce3-4e9f-8834-d79e9cb77047 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:22.292987 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9418ee4d-cce3-4e9f-8834-d79e9cb77047 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:22.298569 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c09fc3e-9327-483d-af4a-5934ffe1d7f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:22.299278 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c09fc3e-9327-483d-af4a-5934ffe1d7f7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:22.299695 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c09fc3e-9327-483d-af4a-5934ffe1d7f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:22.299972 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c09fc3e-9327-483d-af4a-5934ffe1d7f7 admin admin] PATH_INFO: `/v3/projects/7f10a03c88eb45af81a261b82e9b2bd3/users/a89336cd9bc8415f8ea92aafb54e0f55/roles/0487ef7e943e4701915cb8ce0f2899d7` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:22.316031 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c09fc3e-9327-483d-af4a-5934ffe1d7f7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7f10a03c88eb45af81a261b82e9b2bd3, user_id=a89336cd9bc8415f8ea92aafb54e0f55, role_id=0487ef7e943e4701915cb8ce0f2899d7)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:22.319002 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c09fc3e-9327-483d-af4a-5934ffe1d7f7 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:22.346293 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1396/2776] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 01:37:22 2022] PUT /identity/v3/projects/7f10a03c88eb45af81a261b82e9b2bd3/users/a89336cd9bc8415f8ea92aafb54e0f55/roles/0487ef7e943e4701915cb8ce0f2899d7 => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 28 01:37:22.348157 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-074f2951-c85c-471e-abff-008063a7ae0a None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:22.348385 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-074f2951-c85c-471e-abff-008063a7ae0a None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:22.348673 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-074f2951-c85c-471e-abff-008063a7ae0a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:22.377293 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-074f2951-c85c-471e-abff-008063a7ae0a None None] MFA Rules not processed for user `a89336cd9bc8415f8ea92aafb54e0f55`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:37:22.378518 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-074f2951-c85c-471e-abff-008063a7ae0a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:37:22.446292 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1381/2777] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:37:22 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:37:22.458697 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c09fc3e-9327-483d-af4a-5934ffe1d7f7 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:22.461400 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-3c09fc3e-9327-483d-af4a-5934ffe1d7f7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:37:22.496034 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:37:22.531885 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c09fc3e-9327-483d-af4a-5934ffe1d7f7 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:22.537480 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-944e0859-9848-424c-945d-a56edde69a4b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:22.538114 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-944e0859-9848-424c-945d-a56edde69a4b service cinder] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:22.538391 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-944e0859-9848-424c-945d-a56edde69a4b service cinder] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:22.538664 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-944e0859-9848-424c-945d-a56edde69a4b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:22.551209 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-944e0859-9848-424c-945d-a56edde69a4b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:22.552814 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-944e0859-9848-424c-945d-a56edde69a4b service cinder] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:22.565313 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1397/2778] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:37:22 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:37:22.837968 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bcb4b17-3cea-4452-b953-e3afc90289c4 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:22.838362 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bcb4b17-3cea-4452-b953-e3afc90289c4 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:22.838753 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bcb4b17-3cea-4452-b953-e3afc90289c4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:22.874290 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-1bcb4b17-3cea-4452-b953-e3afc90289c4 None None] MFA Rules not processed for user `5317b0b17103487daa2b71273c4ec2c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:37:22.876103 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-1bcb4b17-3cea-4452-b953-e3afc90289c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:37:22.891726 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:37:22.961408 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1382/2779] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:37:22 2022] POST /identity/v3/auth/tokens => generated 3349 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:37:24.043724 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-944e0859-9848-424c-945d-a56edde69a4b service cinder] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:24.137651 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-944e0859-9848-424c-945d-a56edde69a4b service cinder] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:24.142130 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef1dd862-b7b2-41e4-9a1d-dfaf28716a38 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:24.142775 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef1dd862-b7b2-41e4-9a1d-dfaf28716a38 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:24.143046 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef1dd862-b7b2-41e4-9a1d-dfaf28716a38 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:24.143541 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef1dd862-b7b2-41e4-9a1d-dfaf28716a38 admin admin] PATH_INFO: `/v3/users/52a472bb99494401ac5d099b4cfca133` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:24.144424 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef1dd862-b7b2-41e4-9a1d-dfaf28716a38 admin admin] RBAC: Authorizing `identity:delete_user(user_id=52a472bb99494401ac5d099b4cfca133)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:24.146397 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef1dd862-b7b2-41e4-9a1d-dfaf28716a38 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:24.200167 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-ef1dd862-b7b2-41e4-9a1d-dfaf28716a38 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '52a472bb99494401ac5d099b4cfca133'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:24.200810 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-ef1dd862-b7b2-41e4-9a1d-dfaf28716a38 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '52a472bb99494401ac5d099b4cfca133'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:24.209141 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-ef1dd862-b7b2-41e4-9a1d-dfaf28716a38 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '52a472bb99494401ac5d099b4cfca133'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:24.216113 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-ef1dd862-b7b2-41e4-9a1d-dfaf28716a38 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '52a472bb99494401ac5d099b4cfca133'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:24.216747 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-ef1dd862-b7b2-41e4-9a1d-dfaf28716a38 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '52a472bb99494401ac5d099b4cfca133'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:24.234564 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1398/2780] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 01:37:24 2022] DELETE /identity/v3/users/52a472bb99494401ac5d099b4cfca133 => generated 0 bytes in 192 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:37:24.422469 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bcb4b17-3cea-4452-b953-e3afc90289c4 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:24.434164 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bcb4b17-3cea-4452-b953-e3afc90289c4 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:24.502260 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-818a722a-69c7-48f7-9d7c-70e0e848af09 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:24.502952 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-818a722a-69c7-48f7-9d7c-70e0e848af09 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:24.503253 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-818a722a-69c7-48f7-9d7c-70e0e848af09 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:24.503512 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-818a722a-69c7-48f7-9d7c-70e0e848af09 admin admin] PATH_INFO: `/v3/projects/315c865ba0e744059bca147998fd1302` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:24.504927 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-818a722a-69c7-48f7-9d7c-70e0e848af09 admin admin] RBAC: Authorizing `identity:delete_project(project_id=315c865ba0e744059bca147998fd1302)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:24.506928 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-818a722a-69c7-48f7-9d7c-70e0e848af09 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:24.556756 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-818a722a-69c7-48f7-9d7c-70e0e848af09 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '315c865ba0e744059bca147998fd1302', 'request_id': 'req-818a722a-69c7-48f7-9d7c-70e0e848af09'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:24.563498 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-818a722a-69c7-48f7-9d7c-70e0e848af09 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '315c865ba0e744059bca147998fd1302', 'request_id': 'req-818a722a-69c7-48f7-9d7c-70e0e848af09'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:24.570231 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-818a722a-69c7-48f7-9d7c-70e0e848af09 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '315c865ba0e744059bca147998fd1302', 'request_id': 'req-818a722a-69c7-48f7-9d7c-70e0e848af09'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:24.585051 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-818a722a-69c7-48f7-9d7c-70e0e848af09 admin admin] The token cache is being invalidate because project 315c865ba0e744059bca147998fd1302 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81507) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 01:37:24.585315 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-818a722a-69c7-48f7-9d7c-70e0e848af09 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:24.586002 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-818a722a-69c7-48f7-9d7c-70e0e848af09 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:24.587489 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1383/2781] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 01:37:24 2022] DELETE /identity/v3/projects/315c865ba0e744059bca147998fd1302 => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:37:24.591862 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef1dd862-b7b2-41e4-9a1d-dfaf28716a38 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:24.593379 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-ef1dd862-b7b2-41e4-9a1d-dfaf28716a38 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:37:24.607315 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef1dd862-b7b2-41e4-9a1d-dfaf28716a38 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:24.660899 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:37:24.675901 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ab26f07-b362-4635-a23e-7accc595dce2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:24.676474 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ab26f07-b362-4635-a23e-7accc595dce2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:24.676752 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ab26f07-b362-4635-a23e-7accc595dce2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:24.677057 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ab26f07-b362-4635-a23e-7accc595dce2 admin admin] PATH_INFO: `/v3/users/5317b0b17103487daa2b71273c4ec2c0` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:24.678183 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ab26f07-b362-4635-a23e-7accc595dce2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5317b0b17103487daa2b71273c4ec2c0)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:24.680463 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ab26f07-b362-4635-a23e-7accc595dce2 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:24.738710 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-3ab26f07-b362-4635-a23e-7accc595dce2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5317b0b17103487daa2b71273c4ec2c0'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:24.739388 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-3ab26f07-b362-4635-a23e-7accc595dce2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5317b0b17103487daa2b71273c4ec2c0'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:24.749950 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-3ab26f07-b362-4635-a23e-7accc595dce2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5317b0b17103487daa2b71273c4ec2c0'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:24.757720 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-3ab26f07-b362-4635-a23e-7accc595dce2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5317b0b17103487daa2b71273c4ec2c0'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:24.758348 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-3ab26f07-b362-4635-a23e-7accc595dce2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5317b0b17103487daa2b71273c4ec2c0'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:24.775746 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1399/2782] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 01:37:24 2022] DELETE /identity/v3/users/5317b0b17103487daa2b71273c4ec2c0 => generated 0 bytes in 184 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:37:24.959552 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-818a722a-69c7-48f7-9d7c-70e0e848af09 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:24.961099 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-818a722a-69c7-48f7-9d7c-70e0e848af09 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:37:24.979473 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-818a722a-69c7-48f7-9d7c-70e0e848af09 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:25.018617 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:37:25.039126 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2161581-e5d4-4681-8135-611a11d5e22b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:25.039770 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2161581-e5d4-4681-8135-611a11d5e22b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:25.040049 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2161581-e5d4-4681-8135-611a11d5e22b admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:25.040319 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2161581-e5d4-4681-8135-611a11d5e22b admin admin] PATH_INFO: `/v3/projects/239722b27796406ca1638dcdeb05d6ee` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:25.041622 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2161581-e5d4-4681-8135-611a11d5e22b admin admin] RBAC: Authorizing `identity:delete_project(project_id=239722b27796406ca1638dcdeb05d6ee)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:25.043601 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2161581-e5d4-4681-8135-611a11d5e22b admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:25.095714 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-f2161581-e5d4-4681-8135-611a11d5e22b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '239722b27796406ca1638dcdeb05d6ee', 'request_id': 'req-f2161581-e5d4-4681-8135-611a11d5e22b'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:25.101753 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-f2161581-e5d4-4681-8135-611a11d5e22b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '239722b27796406ca1638dcdeb05d6ee', 'request_id': 'req-f2161581-e5d4-4681-8135-611a11d5e22b'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:25.108211 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-f2161581-e5d4-4681-8135-611a11d5e22b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '239722b27796406ca1638dcdeb05d6ee', 'request_id': 'req-f2161581-e5d4-4681-8135-611a11d5e22b'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:25.122212 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-f2161581-e5d4-4681-8135-611a11d5e22b admin admin] The token cache is being invalidate because project 239722b27796406ca1638dcdeb05d6ee was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81507) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 01:37:25.122486 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-f2161581-e5d4-4681-8135-611a11d5e22b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:25.123125 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-f2161581-e5d4-4681-8135-611a11d5e22b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:25.124683 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1384/2783] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 01:37:24 2022] DELETE /identity/v3/projects/239722b27796406ca1638dcdeb05d6ee => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:37:25.129819 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef448bc8-9b59-481e-8794-237a80d37c39 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:25.130257 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef448bc8-9b59-481e-8794-237a80d37c39 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:25.130385 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef448bc8-9b59-481e-8794-237a80d37c39 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:25.154886 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-ef448bc8-9b59-481e-8794-237a80d37c39 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:37:25.156163 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-ef448bc8-9b59-481e-8794-237a80d37c39 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:37:25.216901 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1400/2784] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:37:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:37:25.224523 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8479acfd-2a03-430c-bd4e-871defb59c3f None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:25.224730 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8479acfd-2a03-430c-bd4e-871defb59c3f None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:25.225063 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8479acfd-2a03-430c-bd4e-871defb59c3f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:25.245566 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-8479acfd-2a03-430c-bd4e-871defb59c3f None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:37:25.246798 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-8479acfd-2a03-430c-bd4e-871defb59c3f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:37:25.254961 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1385/2785] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:37:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:37:25.259370 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef448bc8-9b59-481e-8794-237a80d37c39 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:25.269592 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef448bc8-9b59-481e-8794-237a80d37c39 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:25.274505 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cba163b-cb90-438f-aef4-1f2a9618805b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:25.275105 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cba163b-cb90-438f-aef4-1f2a9618805b admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:25.275386 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cba163b-cb90-438f-aef4-1f2a9618805b admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:25.275647 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cba163b-cb90-438f-aef4-1f2a9618805b admin admin] PATH_INFO: `/v3/domains` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:25.276301 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cba163b-cb90-438f-aef4-1f2a9618805b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 01:37:25.276610 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cba163b-cb90-438f-aef4-1f2a9618805b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:25.278322 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cba163b-cb90-438f-aef4-1f2a9618805b admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:25.291080 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1401/2786] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Aug 28 01:37:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 01:37:25.296155 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8479acfd-2a03-430c-bd4e-871defb59c3f None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:25.306902 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8479acfd-2a03-430c-bd4e-871defb59c3f None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:25.312157 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a83a2a70-926f-47b6-ae7e-3e8cedf2e64f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:25.312761 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a83a2a70-926f-47b6-ae7e-3e8cedf2e64f admin admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:25.313062 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a83a2a70-926f-47b6-ae7e-3e8cedf2e64f admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:25.313341 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a83a2a70-926f-47b6-ae7e-3e8cedf2e64f admin admin] PATH_INFO: `/v3/projects` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:25.314004 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a83a2a70-926f-47b6-ae7e-3e8cedf2e64f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:25.315996 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a83a2a70-926f-47b6-ae7e-3e8cedf2e64f admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:25.347097 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1386/2787] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 01:37:25 2022] POST /identity/v3/projects => generated 381 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 01:37:25.352136 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cba163b-cb90-438f-aef4-1f2a9618805b admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:25.365783 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cba163b-cb90-438f-aef4-1f2a9618805b admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:25.420257 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:37:25.434002 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cb0d7e2-76dd-4110-8274-2a4a4a2e7507 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:25.434617 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cb0d7e2-76dd-4110-8274-2a4a4a2e7507 admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:25.434924 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cb0d7e2-76dd-4110-8274-2a4a4a2e7507 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:25.435202 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cb0d7e2-76dd-4110-8274-2a4a4a2e7507 admin admin] PATH_INFO: `/v3/users` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:25.435830 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cb0d7e2-76dd-4110-8274-2a4a4a2e7507 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:25.437988 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cb0d7e2-76dd-4110-8274-2a4a4a2e7507 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:25.460635 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1402/2788] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 01:37:25 2022] POST /identity/v3/users => generated 291 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 01:37:25.465122 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a83a2a70-926f-47b6-ae7e-3e8cedf2e64f admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:25.478030 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a83a2a70-926f-47b6-ae7e-3e8cedf2e64f admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:25.485973 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a4b82ac-842e-45be-968d-41f1f5811b03 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:25.486779 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a4b82ac-842e-45be-968d-41f1f5811b03 admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:25.487051 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a4b82ac-842e-45be-968d-41f1f5811b03 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:25.487476 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a4b82ac-842e-45be-968d-41f1f5811b03 admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:25.488317 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a4b82ac-842e-45be-968d-41f1f5811b03 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:25.490365 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a4b82ac-842e-45be-968d-41f1f5811b03 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:25.503709 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1387/2789] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 01:37:25 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 01:37:25.507999 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cb0d7e2-76dd-4110-8274-2a4a4a2e7507 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:25.518302 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cb0d7e2-76dd-4110-8274-2a4a4a2e7507 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:25.523254 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-948c110f-dd05-4ad7-9dcf-0b12a4554fa0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:25.523849 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-948c110f-dd05-4ad7-9dcf-0b12a4554fa0 admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:25.524128 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-948c110f-dd05-4ad7-9dcf-0b12a4554fa0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:25.524396 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-948c110f-dd05-4ad7-9dcf-0b12a4554fa0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:25.525065 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-948c110f-dd05-4ad7-9dcf-0b12a4554fa0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:25.526683 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-948c110f-dd05-4ad7-9dcf-0b12a4554fa0 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:25.537915 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1403/2790] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 01:37:25 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 01:37:25.542252 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a4b82ac-842e-45be-968d-41f1f5811b03 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:25.552374 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a4b82ac-842e-45be-968d-41f1f5811b03 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:25.558362 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9971b5c9-bced-4bdd-ba74-177418e11f1e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:25.559040 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9971b5c9-bced-4bdd-ba74-177418e11f1e admin admin] REQUEST_METHOD: `PUT` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:25.559317 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9971b5c9-bced-4bdd-ba74-177418e11f1e admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:25.559581 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9971b5c9-bced-4bdd-ba74-177418e11f1e admin admin] PATH_INFO: `/v3/projects/ae92a14fe3cd43e2ae543d875cb014bb/users/bff6fe6e3bd54c9491e38d9c5f66bd52/roles/0487ef7e943e4701915cb8ce0f2899d7` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:25.575657 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9971b5c9-bced-4bdd-ba74-177418e11f1e admin admin] RBAC: Authorizing `identity:create_grant(project_id=ae92a14fe3cd43e2ae543d875cb014bb, user_id=bff6fe6e3bd54c9491e38d9c5f66bd52, role_id=0487ef7e943e4701915cb8ce0f2899d7)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:25.578359 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9971b5c9-bced-4bdd-ba74-177418e11f1e admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:25.597502 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1388/2791] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 01:37:25 2022] PUT /identity/v3/projects/ae92a14fe3cd43e2ae543d875cb014bb/users/bff6fe6e3bd54c9491e38d9c5f66bd52/roles/0487ef7e943e4701915cb8ce0f2899d7 => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 28 01:37:25.608471 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-288aab88-7681-4821-82c2-27d534e2adf3 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:25.608874 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-288aab88-7681-4821-82c2-27d534e2adf3 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:25.609262 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-288aab88-7681-4821-82c2-27d534e2adf3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:25.641335 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-288aab88-7681-4821-82c2-27d534e2adf3 None None] MFA Rules not processed for user `bff6fe6e3bd54c9491e38d9c5f66bd52`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:37:25.642542 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-288aab88-7681-4821-82c2-27d534e2adf3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:37:25.711527 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1404/2792] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:37:25 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:37:25.716221 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9971b5c9-bced-4bdd-ba74-177418e11f1e admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:25.789126 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9971b5c9-bced-4bdd-ba74-177418e11f1e admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:25.797356 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f33f80c2-0ccb-424d-a537-a67395bc29f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:25.797977 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f33f80c2-0ccb-424d-a537-a67395bc29f8 admin admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:25.798260 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f33f80c2-0ccb-424d-a537-a67395bc29f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:25.798535 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f33f80c2-0ccb-424d-a537-a67395bc29f8 admin admin] PATH_INFO: `/v3/projects` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:25.799176 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f33f80c2-0ccb-424d-a537-a67395bc29f8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:25.801169 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f33f80c2-0ccb-424d-a537-a67395bc29f8 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:25.812970 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:37:25.831164 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1389/2793] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 01:37:25 2022] POST /identity/v3/projects => generated 379 bytes in 115 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 01:37:25.835448 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-288aab88-7681-4821-82c2-27d534e2adf3 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:25.846996 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-288aab88-7681-4821-82c2-27d534e2adf3 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:25.909679 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c42b2c0d-ad9a-4e48-af68-1dd6033e4f7a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:25.910457 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c42b2c0d-ad9a-4e48-af68-1dd6033e4f7a admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:25.910782 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c42b2c0d-ad9a-4e48-af68-1dd6033e4f7a admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:25.911077 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c42b2c0d-ad9a-4e48-af68-1dd6033e4f7a admin admin] PATH_INFO: `/v3/users` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:25.911999 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c42b2c0d-ad9a-4e48-af68-1dd6033e4f7a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:25.914628 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c42b2c0d-ad9a-4e48-af68-1dd6033e4f7a admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:25.940387 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1405/2794] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 01:37:25 2022] POST /identity/v3/users => generated 296 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 01:37:25.944646 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f33f80c2-0ccb-424d-a537-a67395bc29f8 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:25.954935 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f33f80c2-0ccb-424d-a537-a67395bc29f8 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:25.959417 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91ffda54-c4e0-4eef-b251-134861174ab7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:25.960014 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91ffda54-c4e0-4eef-b251-134861174ab7 admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:25.960288 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91ffda54-c4e0-4eef-b251-134861174ab7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:25.960578 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91ffda54-c4e0-4eef-b251-134861174ab7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:25.961245 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91ffda54-c4e0-4eef-b251-134861174ab7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:25.962877 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91ffda54-c4e0-4eef-b251-134861174ab7 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:25.973473 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1390/2795] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 01:37:25 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 01:37:25.977994 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c42b2c0d-ad9a-4e48-af68-1dd6033e4f7a admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:25.989432 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c42b2c0d-ad9a-4e48-af68-1dd6033e4f7a admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:25.994682 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3976bf93-5aaa-4b5d-a4c8-e0d6f4cc169d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:25.995338 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3976bf93-5aaa-4b5d-a4c8-e0d6f4cc169d admin admin] REQUEST_METHOD: `PUT` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:25.995617 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3976bf93-5aaa-4b5d-a4c8-e0d6f4cc169d admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:25.995859 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3976bf93-5aaa-4b5d-a4c8-e0d6f4cc169d admin admin] PATH_INFO: `/v3/projects/0a7339daf30840bfbef9eae0ba23c948/users/c250dd3208b34194a967af376b16c940/roles/f08d6043261747e3b4763c744eef7dfb` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:26.017951 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3976bf93-5aaa-4b5d-a4c8-e0d6f4cc169d admin admin] RBAC: Authorizing `identity:create_grant(project_id=0a7339daf30840bfbef9eae0ba23c948, user_id=c250dd3208b34194a967af376b16c940, role_id=f08d6043261747e3b4763c744eef7dfb)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:26.021553 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3976bf93-5aaa-4b5d-a4c8-e0d6f4cc169d admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:26.041805 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1406/2796] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 01:37:25 2022] PUT /identity/v3/projects/0a7339daf30840bfbef9eae0ba23c948/users/c250dd3208b34194a967af376b16c940/roles/f08d6043261747e3b4763c744eef7dfb => generated 0 bytes in 64 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 28 01:37:26.047653 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91ffda54-c4e0-4eef-b251-134861174ab7 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:26.058609 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91ffda54-c4e0-4eef-b251-134861174ab7 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:26.116612 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-116b561d-d88f-4dab-9ed0-95cce38d1a3e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:26.117260 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-116b561d-d88f-4dab-9ed0-95cce38d1a3e admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:26.117544 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-116b561d-d88f-4dab-9ed0-95cce38d1a3e admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:26.117816 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-116b561d-d88f-4dab-9ed0-95cce38d1a3e admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:26.118464 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-116b561d-d88f-4dab-9ed0-95cce38d1a3e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:26.120098 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-116b561d-d88f-4dab-9ed0-95cce38d1a3e admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:26.130919 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1391/2797] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 01:37:26 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 01:37:26.136597 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3976bf93-5aaa-4b5d-a4c8-e0d6f4cc169d admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:26.147801 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3976bf93-5aaa-4b5d-a4c8-e0d6f4cc169d admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:26.153156 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e4ee989-f7e1-4762-9934-83c3827b98ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:26.153805 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e4ee989-f7e1-4762-9934-83c3827b98ac admin admin] REQUEST_METHOD: `PUT` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:26.154079 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e4ee989-f7e1-4762-9934-83c3827b98ac admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:26.154350 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e4ee989-f7e1-4762-9934-83c3827b98ac admin admin] PATH_INFO: `/v3/projects/0a7339daf30840bfbef9eae0ba23c948/users/c250dd3208b34194a967af376b16c940/roles/f08d6043261747e3b4763c744eef7dfb` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:26.156594 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e4ee989-f7e1-4762-9934-83c3827b98ac admin admin] RBAC: Authorizing `identity:create_grant(project_id=0a7339daf30840bfbef9eae0ba23c948, user_id=c250dd3208b34194a967af376b16c940, role_id=f08d6043261747e3b4763c744eef7dfb)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:26.159296 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e4ee989-f7e1-4762-9934-83c3827b98ac admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:26.170304 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1407/2798] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 01:37:26 2022] PUT /identity/v3/projects/0a7339daf30840bfbef9eae0ba23c948/users/c250dd3208b34194a967af376b16c940/roles/f08d6043261747e3b4763c744eef7dfb => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:37:26.182556 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-351f546b-0878-402d-b95c-2daff424831b None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:26.182556 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-351f546b-0878-402d-b95c-2daff424831b None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:26.182764 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-351f546b-0878-402d-b95c-2daff424831b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:26.214518 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-351f546b-0878-402d-b95c-2daff424831b None None] MFA Rules not processed for user `c250dd3208b34194a967af376b16c940`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:37:26.215744 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-351f546b-0878-402d-b95c-2daff424831b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:37:26.295912 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1392/2799] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:37:26 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:37:26.308156 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e4ee989-f7e1-4762-9934-83c3827b98ac admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:26.309962 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-2e4ee989-f7e1-4762-9934-83c3827b98ac admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:37:26.325653 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:37:26.370765 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e4ee989-f7e1-4762-9934-83c3827b98ac admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:26.375397 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf9deb06-5461-4b6d-ad3b-8916dd17aff5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:26.376008 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf9deb06-5461-4b6d-ad3b-8916dd17aff5 service cinder] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:26.376286 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf9deb06-5461-4b6d-ad3b-8916dd17aff5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:26.376556 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf9deb06-5461-4b6d-ad3b-8916dd17aff5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:26.386060 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf9deb06-5461-4b6d-ad3b-8916dd17aff5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:26.387506 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf9deb06-5461-4b6d-ad3b-8916dd17aff5 service cinder] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:26.397491 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1408/2800] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:37:26 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:37:26.485234 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-351f546b-0878-402d-b95c-2daff424831b None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:26.495774 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-351f546b-0878-402d-b95c-2daff424831b None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:26.500267 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc90266e-80e2-48de-976e-1dff59d7bdd8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:26.500912 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc90266e-80e2-48de-976e-1dff59d7bdd8 service cinder] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:26.501211 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc90266e-80e2-48de-976e-1dff59d7bdd8 service cinder] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:26.501500 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc90266e-80e2-48de-976e-1dff59d7bdd8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:26.569956 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc90266e-80e2-48de-976e-1dff59d7bdd8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:26.571874 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc90266e-80e2-48de-976e-1dff59d7bdd8 service cinder] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:26.578225 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:37:26.583217 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1393/2801] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:37:26 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:37:30.794763 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf9deb06-5461-4b6d-ad3b-8916dd17aff5 service cinder] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:30.796626 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-cf9deb06-5461-4b6d-ad3b-8916dd17aff5 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:37:30.856551 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf9deb06-5461-4b6d-ad3b-8916dd17aff5 service cinder] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:30.860528 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-026f280c-a802-4bc9-ab67-b0642f05e61f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:30.861162 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-026f280c-a802-4bc9-ab67-b0642f05e61f service neutron] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:30.861437 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-026f280c-a802-4bc9-ab67-b0642f05e61f service neutron] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:30.861706 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-026f280c-a802-4bc9-ab67-b0642f05e61f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:30.863551 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-026f280c-a802-4bc9-ab67-b0642f05e61f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:37:30.944321 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-026f280c-a802-4bc9-ab67-b0642f05e61f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:30.945805 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-026f280c-a802-4bc9-ab67-b0642f05e61f service neutron] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:30.958159 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1409/2802] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:37:30 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 165 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:37:40.951090 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=70 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:37:43.801983 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc90266e-80e2-48de-976e-1dff59d7bdd8 service cinder] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:43.866699 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc90266e-80e2-48de-976e-1dff59d7bdd8 service cinder] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:43.871578 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d050fc18-b57e-4ca6-b9a1-07755cafb863 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:43.872327 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d050fc18-b57e-4ca6-b9a1-07755cafb863 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:43.872668 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d050fc18-b57e-4ca6-b9a1-07755cafb863 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:43.873007 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d050fc18-b57e-4ca6-b9a1-07755cafb863 admin admin] PATH_INFO: `/v3/users/bff6fe6e3bd54c9491e38d9c5f66bd52` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:43.874269 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d050fc18-b57e-4ca6-b9a1-07755cafb863 admin admin] RBAC: Authorizing `identity:delete_user(user_id=bff6fe6e3bd54c9491e38d9c5f66bd52)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:43.876710 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d050fc18-b57e-4ca6-b9a1-07755cafb863 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:43.925325 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-d050fc18-b57e-4ca6-b9a1-07755cafb863 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'bff6fe6e3bd54c9491e38d9c5f66bd52'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:43.937174 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-d050fc18-b57e-4ca6-b9a1-07755cafb863 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'bff6fe6e3bd54c9491e38d9c5f66bd52'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:43.946553 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-d050fc18-b57e-4ca6-b9a1-07755cafb863 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'bff6fe6e3bd54c9491e38d9c5f66bd52'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:43.953419 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-d050fc18-b57e-4ca6-b9a1-07755cafb863 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'bff6fe6e3bd54c9491e38d9c5f66bd52'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:43.954035 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-d050fc18-b57e-4ca6-b9a1-07755cafb863 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'bff6fe6e3bd54c9491e38d9c5f66bd52'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:43.959793 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1394/2803] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 01:37:43 2022] DELETE /identity/v3/users/bff6fe6e3bd54c9491e38d9c5f66bd52 => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:37:43.964613 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fd598cd-c225-40d1-9301-8e04016a4cf4 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:43.965136 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fd598cd-c225-40d1-9301-8e04016a4cf4 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:43.965230 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fd598cd-c225-40d1-9301-8e04016a4cf4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:43.984917 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-6fd598cd-c225-40d1-9301-8e04016a4cf4 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:37:43.986177 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-6fd598cd-c225-40d1-9301-8e04016a4cf4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:37:44.058256 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1410/2804] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:37:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:37:44.069714 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d050fc18-b57e-4ca6-b9a1-07755cafb863 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:44.071187 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-d050fc18-b57e-4ca6-b9a1-07755cafb863 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:37:44.139152 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d050fc18-b57e-4ca6-b9a1-07755cafb863 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:44.143316 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48016512-e82e-40a9-bd31-6e52acfbe08d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:44.143919 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48016512-e82e-40a9-bd31-6e52acfbe08d service neutron] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:44.144188 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48016512-e82e-40a9-bd31-6e52acfbe08d service neutron] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:44.144467 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48016512-e82e-40a9-bd31-6e52acfbe08d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:44.154070 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48016512-e82e-40a9-bd31-6e52acfbe08d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:44.155525 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48016512-e82e-40a9-bd31-6e52acfbe08d service neutron] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:44.166736 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1395/2805] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:37:44 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:37:44.484254 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fd598cd-c225-40d1-9301-8e04016a4cf4 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:44.485902 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-6fd598cd-c225-40d1-9301-8e04016a4cf4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:37:44.498627 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fd598cd-c225-40d1-9301-8e04016a4cf4 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:44.504443 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e08fe1d2-bdbd-47c1-abf4-79919d266bda admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:44.505309 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e08fe1d2-bdbd-47c1-abf4-79919d266bda admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:44.505591 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e08fe1d2-bdbd-47c1-abf4-79919d266bda admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:44.505866 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e08fe1d2-bdbd-47c1-abf4-79919d266bda admin admin] PATH_INFO: `/v3/projects/ae92a14fe3cd43e2ae543d875cb014bb` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:44.507081 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e08fe1d2-bdbd-47c1-abf4-79919d266bda admin admin] RBAC: Authorizing `identity:delete_project(project_id=ae92a14fe3cd43e2ae543d875cb014bb)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:44.509101 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e08fe1d2-bdbd-47c1-abf4-79919d266bda admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:44.562734 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-e08fe1d2-bdbd-47c1-abf4-79919d266bda admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ae92a14fe3cd43e2ae543d875cb014bb', 'request_id': 'req-e08fe1d2-bdbd-47c1-abf4-79919d266bda'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:44.568370 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-e08fe1d2-bdbd-47c1-abf4-79919d266bda admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ae92a14fe3cd43e2ae543d875cb014bb', 'request_id': 'req-e08fe1d2-bdbd-47c1-abf4-79919d266bda'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:44.574966 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-e08fe1d2-bdbd-47c1-abf4-79919d266bda admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ae92a14fe3cd43e2ae543d875cb014bb', 'request_id': 'req-e08fe1d2-bdbd-47c1-abf4-79919d266bda'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:44.593073 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-e08fe1d2-bdbd-47c1-abf4-79919d266bda admin admin] The token cache is being invalidate because project ae92a14fe3cd43e2ae543d875cb014bb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81508) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 01:37:44.593361 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-e08fe1d2-bdbd-47c1-abf4-79919d266bda admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:44.594393 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-e08fe1d2-bdbd-47c1-abf4-79919d266bda admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:44.596755 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1411/2806] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 01:37:44 2022] DELETE /identity/v3/projects/ae92a14fe3cd43e2ae543d875cb014bb => generated 0 bytes in 113 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:37:44.601230 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48016512-e82e-40a9-bd31-6e52acfbe08d service neutron] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:44.611348 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48016512-e82e-40a9-bd31-6e52acfbe08d service neutron] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:44.631168 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:37:44.675234 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fabd024-af44-4f28-8f0b-6216ca79e241 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:44.675779 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fabd024-af44-4f28-8f0b-6216ca79e241 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:44.676058 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fabd024-af44-4f28-8f0b-6216ca79e241 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:44.676336 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fabd024-af44-4f28-8f0b-6216ca79e241 admin admin] PATH_INFO: `/v3/users/c250dd3208b34194a967af376b16c940` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:44.677510 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fabd024-af44-4f28-8f0b-6216ca79e241 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c250dd3208b34194a967af376b16c940)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:44.679807 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fabd024-af44-4f28-8f0b-6216ca79e241 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:44.731569 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-4fabd024-af44-4f28-8f0b-6216ca79e241 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c250dd3208b34194a967af376b16c940'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:44.744112 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-4fabd024-af44-4f28-8f0b-6216ca79e241 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c250dd3208b34194a967af376b16c940'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:44.754821 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-4fabd024-af44-4f28-8f0b-6216ca79e241 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c250dd3208b34194a967af376b16c940'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:44.766901 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-4fabd024-af44-4f28-8f0b-6216ca79e241 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c250dd3208b34194a967af376b16c940'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:44.767646 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-4fabd024-af44-4f28-8f0b-6216ca79e241 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c250dd3208b34194a967af376b16c940'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:44.773982 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1396/2807] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 01:37:44 2022] DELETE /identity/v3/users/c250dd3208b34194a967af376b16c940 => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:37:45.058757 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e08fe1d2-bdbd-47c1-abf4-79919d266bda admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:45.060230 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-e08fe1d2-bdbd-47c1-abf4-79919d266bda admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:37:45.075895 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e08fe1d2-bdbd-47c1-abf4-79919d266bda admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:45.138627 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-382987cb-9218-403b-80d0-7df6c002e2ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:45.139274 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-382987cb-9218-403b-80d0-7df6c002e2ac admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:45.139544 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-382987cb-9218-403b-80d0-7df6c002e2ac admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:45.139806 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-382987cb-9218-403b-80d0-7df6c002e2ac admin admin] PATH_INFO: `/v3/projects/0a7339daf30840bfbef9eae0ba23c948` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:45.141088 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-382987cb-9218-403b-80d0-7df6c002e2ac admin admin] RBAC: Authorizing `identity:delete_project(project_id=0a7339daf30840bfbef9eae0ba23c948)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:45.143056 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-382987cb-9218-403b-80d0-7df6c002e2ac admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:45.176508 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:37:45.195528 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-382987cb-9218-403b-80d0-7df6c002e2ac admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0a7339daf30840bfbef9eae0ba23c948', 'request_id': 'req-382987cb-9218-403b-80d0-7df6c002e2ac'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:45.201090 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-382987cb-9218-403b-80d0-7df6c002e2ac admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0a7339daf30840bfbef9eae0ba23c948', 'request_id': 'req-382987cb-9218-403b-80d0-7df6c002e2ac'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:45.207442 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-382987cb-9218-403b-80d0-7df6c002e2ac admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0a7339daf30840bfbef9eae0ba23c948', 'request_id': 'req-382987cb-9218-403b-80d0-7df6c002e2ac'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:45.222825 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-382987cb-9218-403b-80d0-7df6c002e2ac admin admin] The token cache is being invalidate because project 0a7339daf30840bfbef9eae0ba23c948 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81508) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 01:37:45.223273 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-382987cb-9218-403b-80d0-7df6c002e2ac admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:45.224016 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-382987cb-9218-403b-80d0-7df6c002e2ac admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:45.225762 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1412/2808] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 01:37:45 2022] DELETE /identity/v3/projects/0a7339daf30840bfbef9eae0ba23c948 => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 28 01:37:45.232177 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63f6b04a-85b8-4bc2-9afc-d8e971f4f69d None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:45.232671 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63f6b04a-85b8-4bc2-9afc-d8e971f4f69d None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:45.232763 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63f6b04a-85b8-4bc2-9afc-d8e971f4f69d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:45.254350 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-63f6b04a-85b8-4bc2-9afc-d8e971f4f69d None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:37:45.255577 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-63f6b04a-85b8-4bc2-9afc-d8e971f4f69d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:37:45.324678 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1397/2809] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:37:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:37:45.336627 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f240a11-3878-4f26-88aa-a3d8e3e3715a None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:45.337615 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f240a11-3878-4f26-88aa-a3d8e3e3715a None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:45.337898 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f240a11-3878-4f26-88aa-a3d8e3e3715a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:45.358765 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-2f240a11-3878-4f26-88aa-a3d8e3e3715a None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:37:45.359988 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-2f240a11-3878-4f26-88aa-a3d8e3e3715a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:37:45.368768 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1413/2810] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:37:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:37:45.373548 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63f6b04a-85b8-4bc2-9afc-d8e971f4f69d None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:45.384006 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63f6b04a-85b8-4bc2-9afc-d8e971f4f69d None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:45.389804 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01e75b2b-19b1-472e-beb5-5c0b56d52636 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:45.390428 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01e75b2b-19b1-472e-beb5-5c0b56d52636 admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:45.390691 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01e75b2b-19b1-472e-beb5-5c0b56d52636 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:45.390964 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01e75b2b-19b1-472e-beb5-5c0b56d52636 admin admin] PATH_INFO: `/v3/domains` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:45.391621 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01e75b2b-19b1-472e-beb5-5c0b56d52636 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 01:37:45.391923 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01e75b2b-19b1-472e-beb5-5c0b56d52636 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:45.393587 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01e75b2b-19b1-472e-beb5-5c0b56d52636 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:45.407073 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1398/2811] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Aug 28 01:37:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 01:37:45.411897 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f240a11-3878-4f26-88aa-a3d8e3e3715a None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:45.421887 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f240a11-3878-4f26-88aa-a3d8e3e3715a None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:45.426739 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-035307a0-d0fc-48fb-8702-c900bffbc919 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:45.427292 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-035307a0-d0fc-48fb-8702-c900bffbc919 admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:45.427718 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-035307a0-d0fc-48fb-8702-c900bffbc919 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:45.427996 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-035307a0-d0fc-48fb-8702-c900bffbc919 admin admin] PATH_INFO: `/v3/projects` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:45.428664 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-035307a0-d0fc-48fb-8702-c900bffbc919 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:45.431023 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-035307a0-d0fc-48fb-8702-c900bffbc919 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:45.461603 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1414/2812] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 01:37:45 2022] POST /identity/v3/projects => generated 363 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 01:37:45.466163 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01e75b2b-19b1-472e-beb5-5c0b56d52636 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:45.476650 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01e75b2b-19b1-472e-beb5-5c0b56d52636 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:45.502477 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:37:45.536852 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-280f6a05-81ed-463f-a1e8-15fad1f7678b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:45.537491 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-280f6a05-81ed-463f-a1e8-15fad1f7678b admin admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:45.537766 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-280f6a05-81ed-463f-a1e8-15fad1f7678b admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:45.538035 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-280f6a05-81ed-463f-a1e8-15fad1f7678b admin admin] PATH_INFO: `/v3/users` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:45.538684 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-280f6a05-81ed-463f-a1e8-15fad1f7678b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:45.540660 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-280f6a05-81ed-463f-a1e8-15fad1f7678b admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:45.571090 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1399/2813] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 01:37:45 2022] POST /identity/v3/users => generated 282 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 01:37:45.575627 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-035307a0-d0fc-48fb-8702-c900bffbc919 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:45.586981 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-035307a0-d0fc-48fb-8702-c900bffbc919 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:45.592166 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c1729ee-48f9-4831-ad9c-fba496d37946 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:45.592779 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c1729ee-48f9-4831-ad9c-fba496d37946 admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:45.593079 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c1729ee-48f9-4831-ad9c-fba496d37946 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:45.593335 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c1729ee-48f9-4831-ad9c-fba496d37946 admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:45.593981 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c1729ee-48f9-4831-ad9c-fba496d37946 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:45.595591 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c1729ee-48f9-4831-ad9c-fba496d37946 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:45.606232 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1415/2814] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 01:37:45 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 01:37:45.612707 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-280f6a05-81ed-463f-a1e8-15fad1f7678b admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:45.626873 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-280f6a05-81ed-463f-a1e8-15fad1f7678b admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:45.633212 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5d37ac3-2200-4218-bc95-b0f95cbb2a9d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:45.633841 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5d37ac3-2200-4218-bc95-b0f95cbb2a9d admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:45.634117 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5d37ac3-2200-4218-bc95-b0f95cbb2a9d admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:45.634373 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5d37ac3-2200-4218-bc95-b0f95cbb2a9d admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:45.635013 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5d37ac3-2200-4218-bc95-b0f95cbb2a9d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:45.636670 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5d37ac3-2200-4218-bc95-b0f95cbb2a9d admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:45.646798 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1400/2815] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 01:37:45 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 01:37:45.651146 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c1729ee-48f9-4831-ad9c-fba496d37946 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:45.661767 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c1729ee-48f9-4831-ad9c-fba496d37946 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:45.667322 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-852c1cfb-c7a9-41f4-8203-6d76ed4914cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:45.667966 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-852c1cfb-c7a9-41f4-8203-6d76ed4914cf admin admin] REQUEST_METHOD: `PUT` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:45.668246 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-852c1cfb-c7a9-41f4-8203-6d76ed4914cf admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:45.668517 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-852c1cfb-c7a9-41f4-8203-6d76ed4914cf admin admin] PATH_INFO: `/v3/projects/8d1758569de247b2bee6fb6f93f61374/users/e2d910a681d946fd99b8dd2717aa0f3c/roles/0487ef7e943e4701915cb8ce0f2899d7` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:45.685368 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-852c1cfb-c7a9-41f4-8203-6d76ed4914cf admin admin] RBAC: Authorizing `identity:create_grant(project_id=8d1758569de247b2bee6fb6f93f61374, user_id=e2d910a681d946fd99b8dd2717aa0f3c, role_id=0487ef7e943e4701915cb8ce0f2899d7)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:45.688240 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-852c1cfb-c7a9-41f4-8203-6d76ed4914cf admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:45.703782 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1416/2816] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 01:37:45 2022] PUT /identity/v3/projects/8d1758569de247b2bee6fb6f93f61374/users/e2d910a681d946fd99b8dd2717aa0f3c/roles/0487ef7e943e4701915cb8ce0f2899d7 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:37:45.715640 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f29e206b-d4de-455c-bc3e-674a560c4a75 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:45.715930 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f29e206b-d4de-455c-bc3e-674a560c4a75 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:45.716214 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f29e206b-d4de-455c-bc3e-674a560c4a75 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:45.749623 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-f29e206b-d4de-455c-bc3e-674a560c4a75 None None] MFA Rules not processed for user `e2d910a681d946fd99b8dd2717aa0f3c`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:37:45.750855 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-f29e206b-d4de-455c-bc3e-674a560c4a75 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:37:45.822979 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1401/2817] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:37:45 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:37:45.827674 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-852c1cfb-c7a9-41f4-8203-6d76ed4914cf admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:45.890952 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-852c1cfb-c7a9-41f4-8203-6d76ed4914cf admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:45.896296 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca4e0372-7ef8-4119-af7f-1fdb3217a553 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:45.896903 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca4e0372-7ef8-4119-af7f-1fdb3217a553 admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:45.897207 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca4e0372-7ef8-4119-af7f-1fdb3217a553 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:45.897481 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca4e0372-7ef8-4119-af7f-1fdb3217a553 admin admin] PATH_INFO: `/v3/projects` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:45.898173 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca4e0372-7ef8-4119-af7f-1fdb3217a553 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:45.900075 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca4e0372-7ef8-4119-af7f-1fdb3217a553 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:45.934014 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1417/2818] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 01:37:45 2022] POST /identity/v3/projects => generated 365 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 01:37:45.935288 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f29e206b-d4de-455c-bc3e-674a560c4a75 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:45.946400 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f29e206b-d4de-455c-bc3e-674a560c4a75 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:46.007467 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02cb7088-5e65-468c-b954-a9df67df78a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:46.008289 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02cb7088-5e65-468c-b954-a9df67df78a8 admin admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:46.008679 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02cb7088-5e65-468c-b954-a9df67df78a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:46.009115 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02cb7088-5e65-468c-b954-a9df67df78a8 admin admin] PATH_INFO: `/v3/users` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:46.010121 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02cb7088-5e65-468c-b954-a9df67df78a8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:46.012745 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02cb7088-5e65-468c-b954-a9df67df78a8 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:46.035184 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1402/2819] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 01:37:45 2022] POST /identity/v3/users => generated 289 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 28 01:37:46.039443 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca4e0372-7ef8-4119-af7f-1fdb3217a553 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:46.050231 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca4e0372-7ef8-4119-af7f-1fdb3217a553 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:46.056271 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3246e245-697b-45a4-a183-f2523b59bd5b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:46.056867 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3246e245-697b-45a4-a183-f2523b59bd5b admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:46.057181 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3246e245-697b-45a4-a183-f2523b59bd5b admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:46.057440 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3246e245-697b-45a4-a183-f2523b59bd5b admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:46.058102 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3246e245-697b-45a4-a183-f2523b59bd5b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:46.059712 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3246e245-697b-45a4-a183-f2523b59bd5b admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:46.070663 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1418/2820] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 01:37:46 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 01:37:46.075829 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02cb7088-5e65-468c-b954-a9df67df78a8 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:46.090967 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02cb7088-5e65-468c-b954-a9df67df78a8 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:46.097329 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d7c3cfb-c1d8-4694-aa77-e1d558c5a5a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:46.097986 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d7c3cfb-c1d8-4694-aa77-e1d558c5a5a5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:46.098260 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d7c3cfb-c1d8-4694-aa77-e1d558c5a5a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:46.098527 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d7c3cfb-c1d8-4694-aa77-e1d558c5a5a5 admin admin] PATH_INFO: `/v3/projects/74a60995eeed40d19e75b232e0391ac0/users/ba86edb617a547a5ad6eff5423eb988a/roles/f08d6043261747e3b4763c744eef7dfb` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:46.116194 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d7c3cfb-c1d8-4694-aa77-e1d558c5a5a5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=74a60995eeed40d19e75b232e0391ac0, user_id=ba86edb617a547a5ad6eff5423eb988a, role_id=f08d6043261747e3b4763c744eef7dfb)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:46.119028 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d7c3cfb-c1d8-4694-aa77-e1d558c5a5a5 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:46.135191 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1403/2821] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 01:37:46 2022] PUT /identity/v3/projects/74a60995eeed40d19e75b232e0391ac0/users/ba86edb617a547a5ad6eff5423eb988a/roles/f08d6043261747e3b4763c744eef7dfb => generated 0 bytes in 60 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:37:46.139548 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3246e245-697b-45a4-a183-f2523b59bd5b admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:46.149805 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3246e245-697b-45a4-a183-f2523b59bd5b admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:46.170795 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:DELETE=6,keystone:SELECT=90,keystone:INSERT=4 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:37:46.211545 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0888a030-25a5-4044-a6f8-1cb736c5662c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:46.212071 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0888a030-25a5-4044-a6f8-1cb736c5662c admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:46.212358 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0888a030-25a5-4044-a6f8-1cb736c5662c admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:46.212609 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0888a030-25a5-4044-a6f8-1cb736c5662c admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:46.213273 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0888a030-25a5-4044-a6f8-1cb736c5662c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:46.214910 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0888a030-25a5-4044-a6f8-1cb736c5662c admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:46.224583 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1419/2822] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 01:37:46 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 01:37:46.230891 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d7c3cfb-c1d8-4694-aa77-e1d558c5a5a5 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:46.236492 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:37:46.242268 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d7c3cfb-c1d8-4694-aa77-e1d558c5a5a5 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:46.247991 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dfa4eaf-8e5b-4285-ab63-243f04b5e1fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:46.248641 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dfa4eaf-8e5b-4285-ab63-243f04b5e1fa admin admin] REQUEST_METHOD: `PUT` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:46.248913 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dfa4eaf-8e5b-4285-ab63-243f04b5e1fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:46.249215 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dfa4eaf-8e5b-4285-ab63-243f04b5e1fa admin admin] PATH_INFO: `/v3/projects/74a60995eeed40d19e75b232e0391ac0/users/ba86edb617a547a5ad6eff5423eb988a/roles/f08d6043261747e3b4763c744eef7dfb` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:46.251161 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dfa4eaf-8e5b-4285-ab63-243f04b5e1fa admin admin] RBAC: Authorizing `identity:create_grant(project_id=74a60995eeed40d19e75b232e0391ac0, user_id=ba86edb617a547a5ad6eff5423eb988a, role_id=f08d6043261747e3b4763c744eef7dfb)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:46.253831 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dfa4eaf-8e5b-4285-ab63-243f04b5e1fa admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:46.264073 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1404/2823] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 01:37:46 2022] PUT /identity/v3/projects/74a60995eeed40d19e75b232e0391ac0/users/ba86edb617a547a5ad6eff5423eb988a/roles/f08d6043261747e3b4763c744eef7dfb => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:37:46.274324 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7c686b7-3a2a-4a70-8525-ddfc4c6ddb1a None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:46.274605 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7c686b7-3a2a-4a70-8525-ddfc4c6ddb1a None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:46.274878 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7c686b7-3a2a-4a70-8525-ddfc4c6ddb1a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:46.302283 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-e7c686b7-3a2a-4a70-8525-ddfc4c6ddb1a None None] MFA Rules not processed for user `ba86edb617a547a5ad6eff5423eb988a`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:37:46.303493 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-e7c686b7-3a2a-4a70-8525-ddfc4c6ddb1a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:37:46.375584 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1420/2824] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:37:46 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:37:46.387504 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dfa4eaf-8e5b-4285-ab63-243f04b5e1fa admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:46.390041 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-9dfa4eaf-8e5b-4285-ab63-243f04b5e1fa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:37:46.449042 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dfa4eaf-8e5b-4285-ab63-243f04b5e1fa admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:46.453339 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8670f647-beb4-4d37-b4a6-e562993f79c1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:46.453946 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8670f647-beb4-4d37-b4a6-e562993f79c1 service cinder] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:46.454221 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8670f647-beb4-4d37-b4a6-e562993f79c1 service cinder] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:46.454494 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8670f647-beb4-4d37-b4a6-e562993f79c1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:46.465256 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8670f647-beb4-4d37-b4a6-e562993f79c1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:46.466678 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8670f647-beb4-4d37-b4a6-e562993f79c1 service cinder] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:46.476654 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1405/2825] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:37:46 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:37:46.566317 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7c686b7-3a2a-4a70-8525-ddfc4c6ddb1a None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:46.577187 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7c686b7-3a2a-4a70-8525-ddfc4c6ddb1a None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:46.581673 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18578366-e31e-479c-91ec-d7cedec86382 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:46.582283 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18578366-e31e-479c-91ec-d7cedec86382 service cinder] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:46.582572 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18578366-e31e-479c-91ec-d7cedec86382 service cinder] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:46.582839 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18578366-e31e-479c-91ec-d7cedec86382 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:46.644339 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18578366-e31e-479c-91ec-d7cedec86382 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:46.645825 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18578366-e31e-479c-91ec-d7cedec86382 service cinder] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:46.655914 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1421/2826] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:37:46 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:37:56.472223 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=30,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:37:56.651496 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=80 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:37:58.179313 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8670f647-beb4-4d37-b4a6-e562993f79c1 service cinder] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:58.180861 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-8670f647-beb4-4d37-b4a6-e562993f79c1 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:37:58.256081 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8670f647-beb4-4d37-b4a6-e562993f79c1 service cinder] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:58.262906 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-174c8810-b1de-4163-86ca-600d857deae6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:58.263567 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-174c8810-b1de-4163-86ca-600d857deae6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:58.263849 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-174c8810-b1de-4163-86ca-600d857deae6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:58.264122 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-174c8810-b1de-4163-86ca-600d857deae6 admin admin] PATH_INFO: `/v3/users/a89336cd9bc8415f8ea92aafb54e0f55` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:58.265366 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-174c8810-b1de-4163-86ca-600d857deae6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a89336cd9bc8415f8ea92aafb54e0f55)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:58.267353 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-174c8810-b1de-4163-86ca-600d857deae6 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:58.321789 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-174c8810-b1de-4163-86ca-600d857deae6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a89336cd9bc8415f8ea92aafb54e0f55'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:58.335398 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-174c8810-b1de-4163-86ca-600d857deae6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a89336cd9bc8415f8ea92aafb54e0f55'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:58.344650 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-174c8810-b1de-4163-86ca-600d857deae6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a89336cd9bc8415f8ea92aafb54e0f55'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:58.352822 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-174c8810-b1de-4163-86ca-600d857deae6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a89336cd9bc8415f8ea92aafb54e0f55'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:58.353534 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-174c8810-b1de-4163-86ca-600d857deae6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a89336cd9bc8415f8ea92aafb54e0f55'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:58.359495 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1406/2827] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 01:37:58 2022] DELETE /identity/v3/users/a89336cd9bc8415f8ea92aafb54e0f55 => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:37:58.364884 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-960576b5-1fcf-4d0d-810c-c1902e95b273 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:58.365399 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-960576b5-1fcf-4d0d-810c-c1902e95b273 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:58.365506 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-960576b5-1fcf-4d0d-810c-c1902e95b273 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:58.388535 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-960576b5-1fcf-4d0d-810c-c1902e95b273 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:37:58.389858 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-960576b5-1fcf-4d0d-810c-c1902e95b273 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:37:58.581960 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1422/2828] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:37:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 219 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:37:58.592284 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-174c8810-b1de-4163-86ca-600d857deae6 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:58.593821 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-174c8810-b1de-4163-86ca-600d857deae6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:37:58.655856 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-174c8810-b1de-4163-86ca-600d857deae6 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:58.659858 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3555a2d1-2efd-4217-93e8-d73fc5b0cec1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:58.660480 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3555a2d1-2efd-4217-93e8-d73fc5b0cec1 service neutron] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:58.660757 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3555a2d1-2efd-4217-93e8-d73fc5b0cec1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:58.661780 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3555a2d1-2efd-4217-93e8-d73fc5b0cec1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:58.672839 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3555a2d1-2efd-4217-93e8-d73fc5b0cec1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:58.674363 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3555a2d1-2efd-4217-93e8-d73fc5b0cec1 service neutron] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:58.685711 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1407/2829] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:37:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:37:58.980140 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-960576b5-1fcf-4d0d-810c-c1902e95b273 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:58.981690 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-960576b5-1fcf-4d0d-810c-c1902e95b273 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:37:58.994203 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-960576b5-1fcf-4d0d-810c-c1902e95b273 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:59.000322 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b54cdab3-20ff-4d8c-bdea-9fe3c563877c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:59.001001 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b54cdab3-20ff-4d8c-bdea-9fe3c563877c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:59.001298 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b54cdab3-20ff-4d8c-bdea-9fe3c563877c admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:59.001546 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b54cdab3-20ff-4d8c-bdea-9fe3c563877c admin admin] PATH_INFO: `/v3/projects/7f10a03c88eb45af81a261b82e9b2bd3` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:59.002872 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b54cdab3-20ff-4d8c-bdea-9fe3c563877c admin admin] RBAC: Authorizing `identity:delete_project(project_id=7f10a03c88eb45af81a261b82e9b2bd3)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:59.004785 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b54cdab3-20ff-4d8c-bdea-9fe3c563877c admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:59.057750 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-b54cdab3-20ff-4d8c-bdea-9fe3c563877c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7f10a03c88eb45af81a261b82e9b2bd3', 'request_id': 'req-b54cdab3-20ff-4d8c-bdea-9fe3c563877c'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:59.063397 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-b54cdab3-20ff-4d8c-bdea-9fe3c563877c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7f10a03c88eb45af81a261b82e9b2bd3', 'request_id': 'req-b54cdab3-20ff-4d8c-bdea-9fe3c563877c'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:59.070229 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-b54cdab3-20ff-4d8c-bdea-9fe3c563877c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7f10a03c88eb45af81a261b82e9b2bd3', 'request_id': 'req-b54cdab3-20ff-4d8c-bdea-9fe3c563877c'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:59.088440 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-b54cdab3-20ff-4d8c-bdea-9fe3c563877c admin admin] The token cache is being invalidate because project 7f10a03c88eb45af81a261b82e9b2bd3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81508) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 01:37:59.088809 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-b54cdab3-20ff-4d8c-bdea-9fe3c563877c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:59.089470 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-b54cdab3-20ff-4d8c-bdea-9fe3c563877c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:59.091016 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1423/2830] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 01:37:58 2022] DELETE /identity/v3/projects/7f10a03c88eb45af81a261b82e9b2bd3 => generated 0 bytes in 111 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:37:59.096719 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0a78605-e427-47d6-95fb-2357fb7f013d None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:59.097043 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0a78605-e427-47d6-95fb-2357fb7f013d None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:59.097494 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0a78605-e427-47d6-95fb-2357fb7f013d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:59.121516 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-b0a78605-e427-47d6-95fb-2357fb7f013d None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:37:59.123196 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-b0a78605-e427-47d6-95fb-2357fb7f013d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:37:59.142270 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:37:59.192829 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1408/2831] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:37:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:37:59.202380 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7216a6a5-b28b-461c-b3a7-1b0a65af054b None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:59.202787 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7216a6a5-b28b-461c-b3a7-1b0a65af054b None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:59.202938 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7216a6a5-b28b-461c-b3a7-1b0a65af054b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:59.226717 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-7216a6a5-b28b-461c-b3a7-1b0a65af054b None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:37:59.227931 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-7216a6a5-b28b-461c-b3a7-1b0a65af054b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:37:59.236864 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1424/2832] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:37:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:37:59.241910 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0a78605-e427-47d6-95fb-2357fb7f013d None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:59.252723 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0a78605-e427-47d6-95fb-2357fb7f013d None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:59.258502 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-666e157a-d809-488d-b9e7-dfe61a076e7c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:59.259160 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-666e157a-d809-488d-b9e7-dfe61a076e7c admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:59.259439 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-666e157a-d809-488d-b9e7-dfe61a076e7c admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:59.259709 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-666e157a-d809-488d-b9e7-dfe61a076e7c admin admin] PATH_INFO: `/v3/domains` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:59.260426 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-666e157a-d809-488d-b9e7-dfe61a076e7c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 01:37:59.260749 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-666e157a-d809-488d-b9e7-dfe61a076e7c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:59.262403 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-666e157a-d809-488d-b9e7-dfe61a076e7c admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:59.275028 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1409/2833] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Aug 28 01:37:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 01:37:59.279971 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7216a6a5-b28b-461c-b3a7-1b0a65af054b None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:59.290384 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7216a6a5-b28b-461c-b3a7-1b0a65af054b None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:59.298006 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43909f32-030f-4e75-8849-7d9399af329a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:59.298006 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43909f32-030f-4e75-8849-7d9399af329a admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:59.298006 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43909f32-030f-4e75-8849-7d9399af329a admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:59.298006 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43909f32-030f-4e75-8849-7d9399af329a admin admin] PATH_INFO: `/v3/projects` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:59.298602 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43909f32-030f-4e75-8849-7d9399af329a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:59.300480 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43909f32-030f-4e75-8849-7d9399af329a admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:59.339666 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1425/2834] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 01:37:59 2022] POST /identity/v3/projects => generated 369 bytes in 60 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 01:37:59.344690 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-666e157a-d809-488d-b9e7-dfe61a076e7c admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:59.353139 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43909f32-030f-4e75-8849-7d9399af329a admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:59.354981 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-43909f32-030f-4e75-8849-7d9399af329a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:37:59.358899 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-666e157a-d809-488d-b9e7-dfe61a076e7c admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:59.422579 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:37:59.432951 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43909f32-030f-4e75-8849-7d9399af329a admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:59.439859 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d78c119-3344-4811-bcbb-172704aedae2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:59.440444 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d78c119-3344-4811-bcbb-172704aedae2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:59.440720 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d78c119-3344-4811-bcbb-172704aedae2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:59.441017 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d78c119-3344-4811-bcbb-172704aedae2 admin admin] PATH_INFO: `/v3/users/e2d910a681d946fd99b8dd2717aa0f3c` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:59.442208 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d78c119-3344-4811-bcbb-172704aedae2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e2d910a681d946fd99b8dd2717aa0f3c)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:59.443853 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1480b47-1b97-4c20-985e-d02b45447d67 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:59.444097 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d78c119-3344-4811-bcbb-172704aedae2 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:59.444477 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1480b47-1b97-4c20-985e-d02b45447d67 admin admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:59.444785 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1480b47-1b97-4c20-985e-d02b45447d67 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:59.445086 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1480b47-1b97-4c20-985e-d02b45447d67 admin admin] PATH_INFO: `/v3/users` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:59.445729 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1480b47-1b97-4c20-985e-d02b45447d67 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:59.447668 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1480b47-1b97-4c20-985e-d02b45447d67 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:59.473938 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1410/2835] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 01:37:59 2022] POST /identity/v3/users => generated 285 bytes in 130 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 01:37:59.480662 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1480b47-1b97-4c20-985e-d02b45447d67 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:59.499595 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1480b47-1b97-4c20-985e-d02b45447d67 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:59.502524 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-4d78c119-3344-4811-bcbb-172704aedae2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e2d910a681d946fd99b8dd2717aa0f3c'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:59.503177 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-4d78c119-3344-4811-bcbb-172704aedae2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e2d910a681d946fd99b8dd2717aa0f3c'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:59.508331 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5782715-f779-428c-a4f7-1941b6756a2a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:59.509478 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5782715-f779-428c-a4f7-1941b6756a2a admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:59.510178 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5782715-f779-428c-a4f7-1941b6756a2a admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:59.510651 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5782715-f779-428c-a4f7-1941b6756a2a admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:59.511954 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-4d78c119-3344-4811-bcbb-172704aedae2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e2d910a681d946fd99b8dd2717aa0f3c'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:59.512146 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5782715-f779-428c-a4f7-1941b6756a2a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:59.515275 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5782715-f779-428c-a4f7-1941b6756a2a admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:59.520292 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-4d78c119-3344-4811-bcbb-172704aedae2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e2d910a681d946fd99b8dd2717aa0f3c'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:59.521055 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-4d78c119-3344-4811-bcbb-172704aedae2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e2d910a681d946fd99b8dd2717aa0f3c'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:37:59.535969 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1411/2836] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 01:37:59 2022] GET /identity/v3/roles => generated 1420 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 01:37:59.540127 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5782715-f779-428c-a4f7-1941b6756a2a admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:59.542560 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-c5782715-f779-428c-a4f7-1941b6756a2a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:37:59.546796 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1426/2837] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 01:37:59 2022] DELETE /identity/v3/users/e2d910a681d946fd99b8dd2717aa0f3c => generated 0 bytes in 194 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 28 01:37:59.551262 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed884ada-0183-42f5-8bf6-7e40fea911f6 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:59.551592 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed884ada-0183-42f5-8bf6-7e40fea911f6 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:59.552430 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed884ada-0183-42f5-8bf6-7e40fea911f6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:59.557760 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5782715-f779-428c-a4f7-1941b6756a2a admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:59.572455 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-ed884ada-0183-42f5-8bf6-7e40fea911f6 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:37:59.573707 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-ed884ada-0183-42f5-8bf6-7e40fea911f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:37:59.627410 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15194e1e-3b6c-4a42-9bfa-420bae1973b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:59.628095 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15194e1e-3b6c-4a42-9bfa-420bae1973b3 admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:59.628378 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15194e1e-3b6c-4a42-9bfa-420bae1973b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:59.628808 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15194e1e-3b6c-4a42-9bfa-420bae1973b3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:59.629638 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15194e1e-3b6c-4a42-9bfa-420bae1973b3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:59.631309 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15194e1e-3b6c-4a42-9bfa-420bae1973b3 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:59.653896 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1412/2838] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 01:37:59 2022] GET /identity/v3/roles => generated 1420 bytes in 111 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Aug 28 01:37:59.654187 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1427/2839] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:37:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:37:59.655656 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15194e1e-3b6c-4a42-9bfa-420bae1973b3 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:59.662447 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed884ada-0183-42f5-8bf6-7e40fea911f6 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:59.663948 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-ed884ada-0183-42f5-8bf6-7e40fea911f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:37:59.667089 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15194e1e-3b6c-4a42-9bfa-420bae1973b3 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:59.672060 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48c322da-0ff5-40cd-b042-034b109eabe0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:59.672731 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48c322da-0ff5-40cd-b042-034b109eabe0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:59.673032 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48c322da-0ff5-40cd-b042-034b109eabe0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:59.673301 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48c322da-0ff5-40cd-b042-034b109eabe0 admin admin] PATH_INFO: `/v3/projects/eef563fbe4b64080bdb44afff3039930/users/2d813500dfa64549a505296aa2d8c16f/roles/0487ef7e943e4701915cb8ce0f2899d7` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:59.693864 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48c322da-0ff5-40cd-b042-034b109eabe0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=eef563fbe4b64080bdb44afff3039930, user_id=2d813500dfa64549a505296aa2d8c16f, role_id=0487ef7e943e4701915cb8ce0f2899d7)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:59.696728 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48c322da-0ff5-40cd-b042-034b109eabe0 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:59.710127 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:37:59.715997 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1413/2840] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 01:37:59 2022] PUT /identity/v3/projects/eef563fbe4b64080bdb44afff3039930/users/2d813500dfa64549a505296aa2d8c16f/roles/0487ef7e943e4701915cb8ce0f2899d7 => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:37:59.729917 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed884ada-0183-42f5-8bf6-7e40fea911f6 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:59.732690 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be925222-9765-4584-82ee-5bf906b1c753 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:59.733198 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be925222-9765-4584-82ee-5bf906b1c753 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:59.733701 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be925222-9765-4584-82ee-5bf906b1c753 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:59.762849 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:37:59.767817 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-be925222-9765-4584-82ee-5bf906b1c753 None None] MFA Rules not processed for user `2d813500dfa64549a505296aa2d8c16f`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:37:59.769269 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-be925222-9765-4584-82ee-5bf906b1c753 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:37:59.789381 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e978d98e-4ed8-4eb3-8f1d-0433d0b32017 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:59.790050 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e978d98e-4ed8-4eb3-8f1d-0433d0b32017 service neutron] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:59.790514 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e978d98e-4ed8-4eb3-8f1d-0433d0b32017 service neutron] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:59.790944 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e978d98e-4ed8-4eb3-8f1d-0433d0b32017 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:59.845969 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1414/2841] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:37:59 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 28 01:37:59.861277 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be925222-9765-4584-82ee-5bf906b1c753 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:37:59.863479 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-be925222-9765-4584-82ee-5bf906b1c753 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:37:59.870570 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e978d98e-4ed8-4eb3-8f1d-0433d0b32017 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:59.872523 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e978d98e-4ed8-4eb3-8f1d-0433d0b32017 service neutron] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:59.884953 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1428/2842] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:37:59 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 223 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 28 01:37:59.936603 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be925222-9765-4584-82ee-5bf906b1c753 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:37:59.940903 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a35c1c62-8750-4494-aae9-ee978cfd57cd service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '14176b54cf6640abb3b1c14e3824d4f1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:37:59.941659 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a35c1c62-8750-4494-aae9-ee978cfd57cd service glance] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:37:59.942135 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a35c1c62-8750-4494-aae9-ee978cfd57cd service glance] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:37:59.942397 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a35c1c62-8750-4494-aae9-ee978cfd57cd service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:37:59.953040 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a35c1c62-8750-4494-aae9-ee978cfd57cd service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:37:59.954490 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a35c1c62-8750-4494-aae9-ee978cfd57cd service glance] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:37:59.964920 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1415/2843] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:37:59 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 28 01:38:00.256368 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e978d98e-4ed8-4eb3-8f1d-0433d0b32017 service neutron] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:38:00.258066 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-e978d98e-4ed8-4eb3-8f1d-0433d0b32017 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:38:00.273940 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e978d98e-4ed8-4eb3-8f1d-0433d0b32017 service neutron] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:38:00.279951 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06936b51-4457-4a6e-bc0a-f922a0bdcb19 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:38:00.280759 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06936b51-4457-4a6e-bc0a-f922a0bdcb19 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:38:00.281127 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06936b51-4457-4a6e-bc0a-f922a0bdcb19 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:38:00.290554 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06936b51-4457-4a6e-bc0a-f922a0bdcb19 admin admin] PATH_INFO: `/v3/projects/8d1758569de247b2bee6fb6f93f61374` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:38:00.292377 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06936b51-4457-4a6e-bc0a-f922a0bdcb19 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8d1758569de247b2bee6fb6f93f61374)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:38:00.295265 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06936b51-4457-4a6e-bc0a-f922a0bdcb19 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:38:00.367464 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-06936b51-4457-4a6e-bc0a-f922a0bdcb19 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8d1758569de247b2bee6fb6f93f61374', 'request_id': 'req-06936b51-4457-4a6e-bc0a-f922a0bdcb19'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:38:00.376644 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-06936b51-4457-4a6e-bc0a-f922a0bdcb19 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8d1758569de247b2bee6fb6f93f61374', 'request_id': 'req-06936b51-4457-4a6e-bc0a-f922a0bdcb19'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:38:00.384501 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-06936b51-4457-4a6e-bc0a-f922a0bdcb19 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8d1758569de247b2bee6fb6f93f61374', 'request_id': 'req-06936b51-4457-4a6e-bc0a-f922a0bdcb19'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:38:00.404266 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-06936b51-4457-4a6e-bc0a-f922a0bdcb19 admin admin] The token cache is being invalidate because project 8d1758569de247b2bee6fb6f93f61374 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81508) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 01:38:00.404730 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-06936b51-4457-4a6e-bc0a-f922a0bdcb19 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:38:00.406173 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-06936b51-4457-4a6e-bc0a-f922a0bdcb19 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:38:00.408697 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1429/2844] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 01:38:00 2022] DELETE /identity/v3/projects/8d1758569de247b2bee6fb6f93f61374 => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:38:00.413789 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a35c1c62-8750-4494-aae9-ee978cfd57cd service glance] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:38:00.430063 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a35c1c62-8750-4494-aae9-ee978cfd57cd service glance] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:38:00.517668 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df2b76f5-1027-4286-b3bb-c3ebfef4a8e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:38:00.518533 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df2b76f5-1027-4286-b3bb-c3ebfef4a8e7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:38:00.518712 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df2b76f5-1027-4286-b3bb-c3ebfef4a8e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:38:00.519015 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df2b76f5-1027-4286-b3bb-c3ebfef4a8e7 admin admin] PATH_INFO: `/v3/users/ba86edb617a547a5ad6eff5423eb988a` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:38:00.520374 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df2b76f5-1027-4286-b3bb-c3ebfef4a8e7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ba86edb617a547a5ad6eff5423eb988a)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:38:00.522770 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df2b76f5-1027-4286-b3bb-c3ebfef4a8e7 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:38:00.580092 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:38:00.593278 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-df2b76f5-1027-4286-b3bb-c3ebfef4a8e7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ba86edb617a547a5ad6eff5423eb988a'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:38:00.607304 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-df2b76f5-1027-4286-b3bb-c3ebfef4a8e7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ba86edb617a547a5ad6eff5423eb988a'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:38:00.616325 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-df2b76f5-1027-4286-b3bb-c3ebfef4a8e7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ba86edb617a547a5ad6eff5423eb988a'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:38:00.624368 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-df2b76f5-1027-4286-b3bb-c3ebfef4a8e7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ba86edb617a547a5ad6eff5423eb988a'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:38:00.625146 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-df2b76f5-1027-4286-b3bb-c3ebfef4a8e7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ba86edb617a547a5ad6eff5423eb988a'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:38:00.631866 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1416/2845] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 01:38:00 2022] DELETE /identity/v3/users/ba86edb617a547a5ad6eff5423eb988a => generated 0 bytes in 219 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:38:00.984451 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06936b51-4457-4a6e-bc0a-f922a0bdcb19 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:38:00.986874 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-06936b51-4457-4a6e-bc0a-f922a0bdcb19 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:38:01.026932 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06936b51-4457-4a6e-bc0a-f922a0bdcb19 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:38:01.102918 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a042937f-6ecb-4d4a-a4f3-aac30a56dabe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:38:01.103561 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a042937f-6ecb-4d4a-a4f3-aac30a56dabe admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:38:01.103835 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a042937f-6ecb-4d4a-a4f3-aac30a56dabe admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:38:01.104101 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a042937f-6ecb-4d4a-a4f3-aac30a56dabe admin admin] PATH_INFO: `/v3/projects/74a60995eeed40d19e75b232e0391ac0` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:38:01.105321 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a042937f-6ecb-4d4a-a4f3-aac30a56dabe admin admin] RBAC: Authorizing `identity:delete_project(project_id=74a60995eeed40d19e75b232e0391ac0)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:38:01.107503 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a042937f-6ecb-4d4a-a4f3-aac30a56dabe admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:38:01.115770 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:38:01.171766 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-a042937f-6ecb-4d4a-a4f3-aac30a56dabe admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '74a60995eeed40d19e75b232e0391ac0', 'request_id': 'req-a042937f-6ecb-4d4a-a4f3-aac30a56dabe'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:38:01.177886 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-a042937f-6ecb-4d4a-a4f3-aac30a56dabe admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '74a60995eeed40d19e75b232e0391ac0', 'request_id': 'req-a042937f-6ecb-4d4a-a4f3-aac30a56dabe'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:38:01.185335 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-a042937f-6ecb-4d4a-a4f3-aac30a56dabe admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '74a60995eeed40d19e75b232e0391ac0', 'request_id': 'req-a042937f-6ecb-4d4a-a4f3-aac30a56dabe'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:38:01.200897 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-a042937f-6ecb-4d4a-a4f3-aac30a56dabe admin admin] The token cache is being invalidate because project 74a60995eeed40d19e75b232e0391ac0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81508) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 01:38:01.201347 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-a042937f-6ecb-4d4a-a4f3-aac30a56dabe admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:38:01.202214 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-a042937f-6ecb-4d4a-a4f3-aac30a56dabe admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:38:01.203872 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1430/2846] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 01:38:00 2022] DELETE /identity/v3/projects/74a60995eeed40d19e75b232e0391ac0 => generated 0 bytes in 221 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:38:01.209281 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a9092f8-6560-4791-a3b3-e7687fae541c None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:38:01.209568 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a9092f8-6560-4791-a3b3-e7687fae541c None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:38:01.209843 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a9092f8-6560-4791-a3b3-e7687fae541c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:38:01.231142 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-7a9092f8-6560-4791-a3b3-e7687fae541c None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:38:01.232393 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-7a9092f8-6560-4791-a3b3-e7687fae541c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:38:01.319002 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1417/2847] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:38:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:38:01.327582 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b718f51a-94f3-4092-a994-cc4d7948d9d1 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:38:01.327861 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b718f51a-94f3-4092-a994-cc4d7948d9d1 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:38:01.328132 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b718f51a-94f3-4092-a994-cc4d7948d9d1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:38:01.349759 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-b718f51a-94f3-4092-a994-cc4d7948d9d1 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:38:01.351185 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-b718f51a-94f3-4092-a994-cc4d7948d9d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:38:01.359949 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1431/2848] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:38:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:38:01.365015 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a9092f8-6560-4791-a3b3-e7687fae541c None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:38:01.376383 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a9092f8-6560-4791-a3b3-e7687fae541c None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:38:01.382188 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45afcd60-36ee-4561-8cf9-4862361adc17 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:38:01.382932 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45afcd60-36ee-4561-8cf9-4862361adc17 admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:38:01.383420 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45afcd60-36ee-4561-8cf9-4862361adc17 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:38:01.383895 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45afcd60-36ee-4561-8cf9-4862361adc17 admin admin] PATH_INFO: `/v3/domains` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:38:01.384877 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45afcd60-36ee-4561-8cf9-4862361adc17 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 01:38:01.385223 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45afcd60-36ee-4561-8cf9-4862361adc17 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:38:01.386907 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45afcd60-36ee-4561-8cf9-4862361adc17 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:38:01.401963 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1418/2849] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Aug 28 01:38:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 01:38:01.406828 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b718f51a-94f3-4092-a994-cc4d7948d9d1 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:38:01.417432 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b718f51a-94f3-4092-a994-cc4d7948d9d1 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:38:01.423935 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c7215ff-bd45-4515-9531-7a3e45ecb63d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:38:01.424579 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c7215ff-bd45-4515-9531-7a3e45ecb63d admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:38:01.424848 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c7215ff-bd45-4515-9531-7a3e45ecb63d admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:38:01.425160 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c7215ff-bd45-4515-9531-7a3e45ecb63d admin admin] PATH_INFO: `/v3/projects` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:38:01.425803 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c7215ff-bd45-4515-9531-7a3e45ecb63d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:38:01.427770 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c7215ff-bd45-4515-9531-7a3e45ecb63d admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:38:01.460998 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1432/2850] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 01:38:01 2022] POST /identity/v3/projects => generated 375 bytes in 55 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 01:38:01.465576 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45afcd60-36ee-4561-8cf9-4862361adc17 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:38:01.476682 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45afcd60-36ee-4561-8cf9-4862361adc17 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:38:01.559748 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08f795ae-26b7-42a8-8621-b1843474c127 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:38:01.561181 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08f795ae-26b7-42a8-8621-b1843474c127 admin admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:38:01.561480 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08f795ae-26b7-42a8-8621-b1843474c127 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:38:01.561982 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08f795ae-26b7-42a8-8621-b1843474c127 admin admin] PATH_INFO: `/v3/users` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:38:01.566341 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08f795ae-26b7-42a8-8621-b1843474c127 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:38:01.566341 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08f795ae-26b7-42a8-8621-b1843474c127 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:38:01.600162 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1419/2851] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 01:38:01 2022] POST /identity/v3/users => generated 288 bytes in 135 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 01:38:01.606728 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c7215ff-bd45-4515-9531-7a3e45ecb63d admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:38:01.619650 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c7215ff-bd45-4515-9531-7a3e45ecb63d admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:38:01.624388 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cb58fea-0e9b-44e1-98d4-4d3c3d21cb4c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:38:01.625421 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cb58fea-0e9b-44e1-98d4-4d3c3d21cb4c admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:38:01.625699 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cb58fea-0e9b-44e1-98d4-4d3c3d21cb4c admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:38:01.625969 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cb58fea-0e9b-44e1-98d4-4d3c3d21cb4c admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:38:01.626615 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cb58fea-0e9b-44e1-98d4-4d3c3d21cb4c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:38:01.628236 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cb58fea-0e9b-44e1-98d4-4d3c3d21cb4c admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:38:01.639351 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1433/2852] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 01:38:01 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 01:38:01.645098 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08f795ae-26b7-42a8-8621-b1843474c127 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:38:01.657443 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08f795ae-26b7-42a8-8621-b1843474c127 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:38:01.663867 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0948741-0126-4548-8293-d296c5f945d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:38:01.664690 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0948741-0126-4548-8293-d296c5f945d5 admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:38:01.665011 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0948741-0126-4548-8293-d296c5f945d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:38:01.665283 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0948741-0126-4548-8293-d296c5f945d5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:38:01.666257 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0948741-0126-4548-8293-d296c5f945d5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:38:01.668257 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0948741-0126-4548-8293-d296c5f945d5 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:38:01.682128 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1420/2853] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 01:38:01 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 01:38:01.687052 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cb58fea-0e9b-44e1-98d4-4d3c3d21cb4c admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:38:01.698470 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cb58fea-0e9b-44e1-98d4-4d3c3d21cb4c admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:38:01.703604 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a209726-6a6c-42b7-bbeb-63ec49c76ceb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:38:01.704287 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a209726-6a6c-42b7-bbeb-63ec49c76ceb admin admin] REQUEST_METHOD: `PUT` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:38:01.704562 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a209726-6a6c-42b7-bbeb-63ec49c76ceb admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:38:01.704833 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a209726-6a6c-42b7-bbeb-63ec49c76ceb admin admin] PATH_INFO: `/v3/projects/c89d0b87627d49d0a097bc5814e929c1/users/e35d8b56e3074bff942cc68b3b7581b3/roles/0487ef7e943e4701915cb8ce0f2899d7` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:38:01.721786 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a209726-6a6c-42b7-bbeb-63ec49c76ceb admin admin] RBAC: Authorizing `identity:create_grant(project_id=c89d0b87627d49d0a097bc5814e929c1, user_id=e35d8b56e3074bff942cc68b3b7581b3, role_id=0487ef7e943e4701915cb8ce0f2899d7)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:38:01.724499 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a209726-6a6c-42b7-bbeb-63ec49c76ceb admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:38:01.741333 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1434/2854] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 01:38:01 2022] PUT /identity/v3/projects/c89d0b87627d49d0a097bc5814e929c1/users/e35d8b56e3074bff942cc68b3b7581b3/roles/0487ef7e943e4701915cb8ce0f2899d7 => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:38:01.753749 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beeb43b4-2c8e-4c2e-b28c-307afd51ad01 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:38:01.754014 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beeb43b4-2c8e-4c2e-b28c-307afd51ad01 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:38:01.754287 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beeb43b4-2c8e-4c2e-b28c-307afd51ad01 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:38:01.786296 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-beeb43b4-2c8e-4c2e-b28c-307afd51ad01 None None] MFA Rules not processed for user `e35d8b56e3074bff942cc68b3b7581b3`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:38:01.787599 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-beeb43b4-2c8e-4c2e-b28c-307afd51ad01 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:38:01.793903 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=91,keystone:DELETE=5,keystone:INSERT=4 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:38:01.877058 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1421/2855] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:38:01 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:38:01.882417 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a209726-6a6c-42b7-bbeb-63ec49c76ceb admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:38:01.957257 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a209726-6a6c-42b7-bbeb-63ec49c76ceb admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:38:01.962658 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-977e2338-6d12-4059-a2c0-98c43a459d0f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:38:01.963285 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-977e2338-6d12-4059-a2c0-98c43a459d0f admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:38:01.963741 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-977e2338-6d12-4059-a2c0-98c43a459d0f admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:38:01.964008 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-977e2338-6d12-4059-a2c0-98c43a459d0f admin admin] PATH_INFO: `/v3/projects` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:38:01.964659 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-977e2338-6d12-4059-a2c0-98c43a459d0f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:38:01.966732 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-977e2338-6d12-4059-a2c0-98c43a459d0f admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:38:01.996716 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1435/2856] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 01:38:01 2022] POST /identity/v3/projects => generated 377 bytes in 115 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 01:38:02.001185 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beeb43b4-2c8e-4c2e-b28c-307afd51ad01 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:38:02.012815 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beeb43b4-2c8e-4c2e-b28c-307afd51ad01 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:38:02.082500 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e35541b-e02b-436d-9210-0027572226da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:38:02.083374 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e35541b-e02b-436d-9210-0027572226da admin admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:38:02.083713 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e35541b-e02b-436d-9210-0027572226da admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:38:02.084035 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e35541b-e02b-436d-9210-0027572226da admin admin] PATH_INFO: `/v3/users` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:38:02.084681 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e35541b-e02b-436d-9210-0027572226da admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:38:02.086878 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e35541b-e02b-436d-9210-0027572226da admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:38:02.115340 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1422/2857] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 01:38:02 2022] POST /identity/v3/users => generated 295 bytes in 115 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 01:38:02.121706 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-977e2338-6d12-4059-a2c0-98c43a459d0f admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:38:02.137969 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-977e2338-6d12-4059-a2c0-98c43a459d0f admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:38:02.146084 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-002be722-7961-47e6-ac85-952bb6451f4d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:38:02.147155 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-002be722-7961-47e6-ac85-952bb6451f4d admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:38:02.147668 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-002be722-7961-47e6-ac85-952bb6451f4d admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:38:02.148293 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-002be722-7961-47e6-ac85-952bb6451f4d admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:38:02.149548 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-002be722-7961-47e6-ac85-952bb6451f4d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:38:02.152652 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-002be722-7961-47e6-ac85-952bb6451f4d admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:38:02.173474 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1436/2858] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 01:38:02 2022] GET /identity/v3/roles => generated 1420 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 01:38:02.178117 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e35541b-e02b-436d-9210-0027572226da admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:38:02.189376 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e35541b-e02b-436d-9210-0027572226da admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:38:02.195130 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a18e6c92-463b-4982-870c-757e288b80e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:38:02.195870 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a18e6c92-463b-4982-870c-757e288b80e5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:38:02.196149 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a18e6c92-463b-4982-870c-757e288b80e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:38:02.196417 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a18e6c92-463b-4982-870c-757e288b80e5 admin admin] PATH_INFO: `/v3/projects/2b427d0b77174a98bf107b8f677d7369/users/f690d3b78dcf4c6192db917a0c9d11d7/roles/f08d6043261747e3b4763c744eef7dfb` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:38:02.214306 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a18e6c92-463b-4982-870c-757e288b80e5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2b427d0b77174a98bf107b8f677d7369, user_id=f690d3b78dcf4c6192db917a0c9d11d7, role_id=f08d6043261747e3b4763c744eef7dfb)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:38:02.217026 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a18e6c92-463b-4982-870c-757e288b80e5 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:38:02.234596 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1423/2859] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 01:38:02 2022] PUT /identity/v3/projects/2b427d0b77174a98bf107b8f677d7369/users/f690d3b78dcf4c6192db917a0c9d11d7/roles/f08d6043261747e3b4763c744eef7dfb => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:38:02.239696 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-002be722-7961-47e6-ac85-952bb6451f4d admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:38:02.247081 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:38:02.251374 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-002be722-7961-47e6-ac85-952bb6451f4d admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:38:02.320398 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-747256da-25b0-440c-a321-7eb961dab084 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:38:02.321220 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-747256da-25b0-440c-a321-7eb961dab084 admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:38:02.322195 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-747256da-25b0-440c-a321-7eb961dab084 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:38:02.322570 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-747256da-25b0-440c-a321-7eb961dab084 admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:38:02.323981 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-747256da-25b0-440c-a321-7eb961dab084 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:38:02.326774 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-747256da-25b0-440c-a321-7eb961dab084 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:38:02.347237 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1437/2860] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 01:38:02 2022] GET /identity/v3/roles => generated 1420 bytes in 109 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 01:38:02.354093 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a18e6c92-463b-4982-870c-757e288b80e5 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:38:02.365897 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a18e6c92-463b-4982-870c-757e288b80e5 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:38:02.371647 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa8b53a9-0d8d-4634-921b-d72d05e01bd1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:38:02.372800 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa8b53a9-0d8d-4634-921b-d72d05e01bd1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:38:02.373343 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa8b53a9-0d8d-4634-921b-d72d05e01bd1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:38:02.373838 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa8b53a9-0d8d-4634-921b-d72d05e01bd1 admin admin] PATH_INFO: `/v3/projects/2b427d0b77174a98bf107b8f677d7369/users/f690d3b78dcf4c6192db917a0c9d11d7/roles/f08d6043261747e3b4763c744eef7dfb` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:38:02.377572 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa8b53a9-0d8d-4634-921b-d72d05e01bd1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2b427d0b77174a98bf107b8f677d7369, user_id=f690d3b78dcf4c6192db917a0c9d11d7, role_id=f08d6043261747e3b4763c744eef7dfb)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:38:02.381885 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa8b53a9-0d8d-4634-921b-d72d05e01bd1 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:38:02.394755 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1424/2861] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 01:38:02 2022] PUT /identity/v3/projects/2b427d0b77174a98bf107b8f677d7369/users/f690d3b78dcf4c6192db917a0c9d11d7/roles/f08d6043261747e3b4763c744eef7dfb => generated 0 bytes in 42 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:38:02.406349 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74447f74-de88-48e9-bf37-93327c723ce0 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:38:02.406800 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74447f74-de88-48e9-bf37-93327c723ce0 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:38:02.407156 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74447f74-de88-48e9-bf37-93327c723ce0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:38:02.440888 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-74447f74-de88-48e9-bf37-93327c723ce0 None None] MFA Rules not processed for user `f690d3b78dcf4c6192db917a0c9d11d7`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:38:02.442177 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-74447f74-de88-48e9-bf37-93327c723ce0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:38:02.521726 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1438/2862] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:38:02 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:38:02.538225 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa8b53a9-0d8d-4634-921b-d72d05e01bd1 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:38:02.540277 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-aa8b53a9-0d8d-4634-921b-d72d05e01bd1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:38:02.623183 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa8b53a9-0d8d-4634-921b-d72d05e01bd1 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:38:02.627131 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7598489-c3a4-4096-b980-482ff05ef54d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:38:02.627763 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7598489-c3a4-4096-b980-482ff05ef54d service cinder] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:38:02.628053 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7598489-c3a4-4096-b980-482ff05ef54d service cinder] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:38:02.628314 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7598489-c3a4-4096-b980-482ff05ef54d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:38:02.645033 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:38:02.720159 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7598489-c3a4-4096-b980-482ff05ef54d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:38:02.722004 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7598489-c3a4-4096-b980-482ff05ef54d service cinder] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:38:02.732485 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1425/2863] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:38:02 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 195 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:38:12.518130 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=61 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:38:12.727769 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=21 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:38:18.845225 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74447f74-de88-48e9-bf37-93327c723ce0 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:38:18.858831 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74447f74-de88-48e9-bf37-93327c723ce0 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:38:18.865290 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7101bd2f-16f1-4853-884c-cc8965327317 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:38:18.865897 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7101bd2f-16f1-4853-884c-cc8965327317 service cinder] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:38:18.866173 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7101bd2f-16f1-4853-884c-cc8965327317 service cinder] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:38:18.866427 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7101bd2f-16f1-4853-884c-cc8965327317 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:38:18.876720 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7101bd2f-16f1-4853-884c-cc8965327317 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:38:18.878240 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7101bd2f-16f1-4853-884c-cc8965327317 service cinder] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:38:18.889048 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1439/2864] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:38:18 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:38:28.884229 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=6 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:39:00.505512 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7598489-c3a4-4096-b980-482ff05ef54d service cinder] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:39:00.570288 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7598489-c3a4-4096-b980-482ff05ef54d service cinder] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:39:00.574713 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1da3c36-4c58-49ac-b8e6-a698ce93b69e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:39:00.575404 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1da3c36-4c58-49ac-b8e6-a698ce93b69e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:39:00.575683 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1da3c36-4c58-49ac-b8e6-a698ce93b69e admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:39:00.576029 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1da3c36-4c58-49ac-b8e6-a698ce93b69e admin admin] PATH_INFO: `/v3/users/e35d8b56e3074bff942cc68b3b7581b3` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:39:00.577114 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1da3c36-4c58-49ac-b8e6-a698ce93b69e admin admin] RBAC: Authorizing `identity:delete_user(user_id=e35d8b56e3074bff942cc68b3b7581b3)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:39:00.579079 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1da3c36-4c58-49ac-b8e6-a698ce93b69e admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:39:00.629819 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-c1da3c36-4c58-49ac-b8e6-a698ce93b69e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e35d8b56e3074bff942cc68b3b7581b3'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:39:00.642009 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-c1da3c36-4c58-49ac-b8e6-a698ce93b69e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e35d8b56e3074bff942cc68b3b7581b3'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:39:00.651300 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-c1da3c36-4c58-49ac-b8e6-a698ce93b69e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e35d8b56e3074bff942cc68b3b7581b3'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:39:00.658170 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-c1da3c36-4c58-49ac-b8e6-a698ce93b69e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e35d8b56e3074bff942cc68b3b7581b3'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:39:00.658810 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-c1da3c36-4c58-49ac-b8e6-a698ce93b69e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e35d8b56e3074bff942cc68b3b7581b3'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:39:00.666275 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1426/2865] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 01:39:00 2022] DELETE /identity/v3/users/e35d8b56e3074bff942cc68b3b7581b3 => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:39:00.673824 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c237208e-a03c-4930-accf-26339fe72e44 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:39:00.673824 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c237208e-a03c-4930-accf-26339fe72e44 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:39:00.676319 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c237208e-a03c-4930-accf-26339fe72e44 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:39:00.696206 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-c237208e-a03c-4930-accf-26339fe72e44 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:39:00.697464 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-c237208e-a03c-4930-accf-26339fe72e44 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:39:00.759785 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1440/2866] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:39:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:39:00.770632 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1da3c36-4c58-49ac-b8e6-a698ce93b69e admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:39:00.772171 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-c1da3c36-4c58-49ac-b8e6-a698ce93b69e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:39:00.840998 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1da3c36-4c58-49ac-b8e6-a698ce93b69e admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:39:00.845261 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5ff5bff-6146-4f53-b9f3-99804226144c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:39:00.845878 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5ff5bff-6146-4f53-b9f3-99804226144c service neutron] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:39:00.846164 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5ff5bff-6146-4f53-b9f3-99804226144c service neutron] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:39:00.846436 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5ff5bff-6146-4f53-b9f3-99804226144c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:39:00.856054 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5ff5bff-6146-4f53-b9f3-99804226144c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:39:00.857577 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5ff5bff-6146-4f53-b9f3-99804226144c service neutron] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:39:00.867303 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1427/2867] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:39:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:39:01.159826 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c237208e-a03c-4930-accf-26339fe72e44 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:39:01.161509 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-c237208e-a03c-4930-accf-26339fe72e44 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:39:01.175542 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c237208e-a03c-4930-accf-26339fe72e44 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:39:01.182777 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7d9b073-f85c-4399-821e-b3e69da02fe7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:39:01.183596 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7d9b073-f85c-4399-821e-b3e69da02fe7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:39:01.184004 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7d9b073-f85c-4399-821e-b3e69da02fe7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:39:01.184117 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7d9b073-f85c-4399-821e-b3e69da02fe7 admin admin] PATH_INFO: `/v3/projects/c89d0b87627d49d0a097bc5814e929c1` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:39:01.185370 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7d9b073-f85c-4399-821e-b3e69da02fe7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c89d0b87627d49d0a097bc5814e929c1)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:39:01.187379 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7d9b073-f85c-4399-821e-b3e69da02fe7 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:39:01.243293 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-f7d9b073-f85c-4399-821e-b3e69da02fe7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c89d0b87627d49d0a097bc5814e929c1', 'request_id': 'req-f7d9b073-f85c-4399-821e-b3e69da02fe7'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:39:01.249457 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-f7d9b073-f85c-4399-821e-b3e69da02fe7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c89d0b87627d49d0a097bc5814e929c1', 'request_id': 'req-f7d9b073-f85c-4399-821e-b3e69da02fe7'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:39:01.255960 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-f7d9b073-f85c-4399-821e-b3e69da02fe7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c89d0b87627d49d0a097bc5814e929c1', 'request_id': 'req-f7d9b073-f85c-4399-821e-b3e69da02fe7'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:39:01.272775 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-f7d9b073-f85c-4399-821e-b3e69da02fe7 admin admin] The token cache is being invalidate because project c89d0b87627d49d0a097bc5814e929c1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81508) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 01:39:01.273092 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-f7d9b073-f85c-4399-821e-b3e69da02fe7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:39:01.273745 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-f7d9b073-f85c-4399-821e-b3e69da02fe7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:39:01.275252 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1441/2868] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 01:39:01 2022] DELETE /identity/v3/projects/c89d0b87627d49d0a097bc5814e929c1 => generated 0 bytes in 116 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:39:01.282900 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5ff5bff-6146-4f53-b9f3-99804226144c service neutron] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:39:01.297290 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5ff5bff-6146-4f53-b9f3-99804226144c service neutron] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:39:01.320380 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:39:01.364262 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ac9f9da-20b9-4f75-8f7e-00e9e0e30cf2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:39:01.364914 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ac9f9da-20b9-4f75-8f7e-00e9e0e30cf2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:39:01.365318 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ac9f9da-20b9-4f75-8f7e-00e9e0e30cf2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:39:01.365486 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ac9f9da-20b9-4f75-8f7e-00e9e0e30cf2 admin admin] PATH_INFO: `/v3/users/f690d3b78dcf4c6192db917a0c9d11d7` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:39:01.366678 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ac9f9da-20b9-4f75-8f7e-00e9e0e30cf2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f690d3b78dcf4c6192db917a0c9d11d7)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:39:01.368623 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ac9f9da-20b9-4f75-8f7e-00e9e0e30cf2 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:39:01.430925 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-6ac9f9da-20b9-4f75-8f7e-00e9e0e30cf2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f690d3b78dcf4c6192db917a0c9d11d7'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:39:01.443031 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-6ac9f9da-20b9-4f75-8f7e-00e9e0e30cf2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f690d3b78dcf4c6192db917a0c9d11d7'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:39:01.452230 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-6ac9f9da-20b9-4f75-8f7e-00e9e0e30cf2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f690d3b78dcf4c6192db917a0c9d11d7'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:39:01.460085 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-6ac9f9da-20b9-4f75-8f7e-00e9e0e30cf2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f690d3b78dcf4c6192db917a0c9d11d7'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:39:01.460832 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-6ac9f9da-20b9-4f75-8f7e-00e9e0e30cf2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f690d3b78dcf4c6192db917a0c9d11d7'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:39:01.467207 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1428/2869] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 01:39:01 2022] DELETE /identity/v3/users/f690d3b78dcf4c6192db917a0c9d11d7 => generated 0 bytes in 185 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 28 01:39:01.765977 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7d9b073-f85c-4399-821e-b3e69da02fe7 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:39:01.767624 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-f7d9b073-f85c-4399-821e-b3e69da02fe7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:39:01.783503 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7d9b073-f85c-4399-821e-b3e69da02fe7 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:39:01.869727 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d89fcd92-9d77-4f9a-b522-89463a4b26a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:39:01.870527 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d89fcd92-9d77-4f9a-b522-89463a4b26a3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:39:01.870809 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d89fcd92-9d77-4f9a-b522-89463a4b26a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:39:01.872685 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d89fcd92-9d77-4f9a-b522-89463a4b26a3 admin admin] PATH_INFO: `/v3/projects/2b427d0b77174a98bf107b8f677d7369` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:39:01.874650 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d89fcd92-9d77-4f9a-b522-89463a4b26a3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2b427d0b77174a98bf107b8f677d7369)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:39:01.876649 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d89fcd92-9d77-4f9a-b522-89463a4b26a3 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:39:01.909074 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:39:01.927175 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-d89fcd92-9d77-4f9a-b522-89463a4b26a3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2b427d0b77174a98bf107b8f677d7369', 'request_id': 'req-d89fcd92-9d77-4f9a-b522-89463a4b26a3'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:39:01.932843 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-d89fcd92-9d77-4f9a-b522-89463a4b26a3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2b427d0b77174a98bf107b8f677d7369', 'request_id': 'req-d89fcd92-9d77-4f9a-b522-89463a4b26a3'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:39:01.939147 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-d89fcd92-9d77-4f9a-b522-89463a4b26a3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2b427d0b77174a98bf107b8f677d7369', 'request_id': 'req-d89fcd92-9d77-4f9a-b522-89463a4b26a3'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:39:01.954009 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-d89fcd92-9d77-4f9a-b522-89463a4b26a3 admin admin] The token cache is being invalidate because project 2b427d0b77174a98bf107b8f677d7369 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81508) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 01:39:01.954300 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-d89fcd92-9d77-4f9a-b522-89463a4b26a3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:39:01.954978 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-d89fcd92-9d77-4f9a-b522-89463a4b26a3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:39:01.956482 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1442/2870] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 01:39:01 2022] DELETE /identity/v3/projects/2b427d0b77174a98bf107b8f677d7369 => generated 0 bytes in 191 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:39:01.961703 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b76ab94-bfd7-42f2-99cb-be0d585cd773 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:39:01.962224 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b76ab94-bfd7-42f2-99cb-be0d585cd773 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:39:01.962224 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b76ab94-bfd7-42f2-99cb-be0d585cd773 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:39:01.982752 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-1b76ab94-bfd7-42f2-99cb-be0d585cd773 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:39:01.983967 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-1b76ab94-bfd7-42f2-99cb-be0d585cd773 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:39:02.047973 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1429/2871] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:39:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:39:02.059469 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18af0f52-b4fe-48dc-8832-adfb07db7475 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:39:02.059753 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18af0f52-b4fe-48dc-8832-adfb07db7475 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:39:02.060027 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18af0f52-b4fe-48dc-8832-adfb07db7475 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:39:02.083341 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-18af0f52-b4fe-48dc-8832-adfb07db7475 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:39:02.085010 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-18af0f52-b4fe-48dc-8832-adfb07db7475 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:39:02.096555 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1443/2872] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:39:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:39:02.101158 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b76ab94-bfd7-42f2-99cb-be0d585cd773 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:39:02.111892 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b76ab94-bfd7-42f2-99cb-be0d585cd773 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:39:02.118027 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc0bb49c-0733-4123-9bb5-c7e9056b4bb4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:39:02.118744 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc0bb49c-0733-4123-9bb5-c7e9056b4bb4 admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:39:02.119086 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc0bb49c-0733-4123-9bb5-c7e9056b4bb4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:39:02.119360 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc0bb49c-0733-4123-9bb5-c7e9056b4bb4 admin admin] PATH_INFO: `/v3/domains` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:39:02.120045 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc0bb49c-0733-4123-9bb5-c7e9056b4bb4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 01:39:02.120350 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc0bb49c-0733-4123-9bb5-c7e9056b4bb4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:39:02.122018 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc0bb49c-0733-4123-9bb5-c7e9056b4bb4 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:39:02.134719 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1430/2873] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Aug 28 01:39:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 01:39:02.139352 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18af0f52-b4fe-48dc-8832-adfb07db7475 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:39:02.149099 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18af0f52-b4fe-48dc-8832-adfb07db7475 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:39:02.154160 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c21113b-440c-4878-b274-31a45add27e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:39:02.154869 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c21113b-440c-4878-b274-31a45add27e4 admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:39:02.155152 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c21113b-440c-4878-b274-31a45add27e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:39:02.155439 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c21113b-440c-4878-b274-31a45add27e4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:39:02.156097 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c21113b-440c-4878-b274-31a45add27e4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:39:02.158064 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c21113b-440c-4878-b274-31a45add27e4 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:39:02.188506 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1444/2874] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 01:39:02 2022] POST /identity/v3/projects => generated 369 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 01:39:02.194873 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc0bb49c-0733-4123-9bb5-c7e9056b4bb4 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:39:02.207909 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc0bb49c-0733-4123-9bb5-c7e9056b4bb4 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:39:02.235231 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:39:02.270386 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa5a25ee-3973-425c-926a-376563f41b04 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:39:02.270963 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa5a25ee-3973-425c-926a-376563f41b04 admin admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:39:02.271235 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa5a25ee-3973-425c-926a-376563f41b04 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:39:02.271526 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa5a25ee-3973-425c-926a-376563f41b04 admin admin] PATH_INFO: `/v3/users` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:39:02.272151 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa5a25ee-3973-425c-926a-376563f41b04 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:39:02.274137 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa5a25ee-3973-425c-926a-376563f41b04 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:39:02.296514 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1431/2875] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 01:39:02 2022] POST /identity/v3/users => generated 285 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 01:39:02.301212 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c21113b-440c-4878-b274-31a45add27e4 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:39:02.313018 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c21113b-440c-4878-b274-31a45add27e4 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:39:02.318647 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5025fd92-5677-4cdf-8ca5-22472c69118a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:39:02.319264 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5025fd92-5677-4cdf-8ca5-22472c69118a admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:39:02.319568 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5025fd92-5677-4cdf-8ca5-22472c69118a admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:39:02.319836 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5025fd92-5677-4cdf-8ca5-22472c69118a admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:39:02.320477 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5025fd92-5677-4cdf-8ca5-22472c69118a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:39:02.322212 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5025fd92-5677-4cdf-8ca5-22472c69118a admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:39:02.333371 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1445/2876] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 01:39:02 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 01:39:02.337803 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa5a25ee-3973-425c-926a-376563f41b04 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:39:02.348376 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa5a25ee-3973-425c-926a-376563f41b04 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:39:02.353571 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5fed78d-a999-49c8-937d-8a0e43960e59 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:39:02.354176 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5fed78d-a999-49c8-937d-8a0e43960e59 admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:39:02.354453 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5fed78d-a999-49c8-937d-8a0e43960e59 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:39:02.354724 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5fed78d-a999-49c8-937d-8a0e43960e59 admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:39:02.355370 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5fed78d-a999-49c8-937d-8a0e43960e59 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:39:02.356974 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5fed78d-a999-49c8-937d-8a0e43960e59 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:39:02.367145 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1432/2877] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 01:39:02 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 01:39:02.372550 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5025fd92-5677-4cdf-8ca5-22472c69118a admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:39:02.382662 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5025fd92-5677-4cdf-8ca5-22472c69118a admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:39:02.388127 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf8d41a8-17f7-42c7-89e7-010d148e2aa1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:39:02.388773 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf8d41a8-17f7-42c7-89e7-010d148e2aa1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:39:02.389137 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf8d41a8-17f7-42c7-89e7-010d148e2aa1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:39:02.389406 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf8d41a8-17f7-42c7-89e7-010d148e2aa1 admin admin] PATH_INFO: `/v3/projects/83aa7ef80a3b4a259611f6f684644211/users/7b516f3b75084aed874fa74c7f5ec007/roles/0487ef7e943e4701915cb8ce0f2899d7` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:39:02.405288 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf8d41a8-17f7-42c7-89e7-010d148e2aa1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=83aa7ef80a3b4a259611f6f684644211, user_id=7b516f3b75084aed874fa74c7f5ec007, role_id=0487ef7e943e4701915cb8ce0f2899d7)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:39:02.407956 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf8d41a8-17f7-42c7-89e7-010d148e2aa1 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:39:02.424505 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1446/2878] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 01:39:02 2022] PUT /identity/v3/projects/83aa7ef80a3b4a259611f6f684644211/users/7b516f3b75084aed874fa74c7f5ec007/roles/0487ef7e943e4701915cb8ce0f2899d7 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:39:02.435518 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5cf56f7-533c-42d2-a6ec-ba5089249af2 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:39:02.435834 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5cf56f7-533c-42d2-a6ec-ba5089249af2 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:39:02.436119 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5cf56f7-533c-42d2-a6ec-ba5089249af2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:39:02.467350 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-c5cf56f7-533c-42d2-a6ec-ba5089249af2 None None] MFA Rules not processed for user `7b516f3b75084aed874fa74c7f5ec007`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:39:02.468603 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-c5cf56f7-533c-42d2-a6ec-ba5089249af2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:39:02.543850 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1433/2879] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:39:02 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:39:02.549932 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf8d41a8-17f7-42c7-89e7-010d148e2aa1 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:39:02.613686 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf8d41a8-17f7-42c7-89e7-010d148e2aa1 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:39:02.620241 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e0cf568-9750-4b07-8837-fe9fcdd9d7e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:39:02.620847 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e0cf568-9750-4b07-8837-fe9fcdd9d7e2 admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:39:02.621144 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e0cf568-9750-4b07-8837-fe9fcdd9d7e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:39:02.621451 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e0cf568-9750-4b07-8837-fe9fcdd9d7e2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:39:02.622125 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e0cf568-9750-4b07-8837-fe9fcdd9d7e2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:39:02.624145 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e0cf568-9750-4b07-8837-fe9fcdd9d7e2 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:39:02.659092 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1447/2880] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 01:39:02 2022] POST /identity/v3/projects => generated 367 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 01:39:02.663362 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5cf56f7-533c-42d2-a6ec-ba5089249af2 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:39:02.674699 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5cf56f7-533c-42d2-a6ec-ba5089249af2 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:39:02.740294 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-329b51d0-f488-4b2c-9e1e-cbcd7dd5d48e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:39:02.740889 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-329b51d0-f488-4b2c-9e1e-cbcd7dd5d48e admin admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:39:02.741208 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-329b51d0-f488-4b2c-9e1e-cbcd7dd5d48e admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:39:02.741469 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-329b51d0-f488-4b2c-9e1e-cbcd7dd5d48e admin admin] PATH_INFO: `/v3/users` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:39:02.742121 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-329b51d0-f488-4b2c-9e1e-cbcd7dd5d48e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:39:02.744055 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-329b51d0-f488-4b2c-9e1e-cbcd7dd5d48e admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:39:02.766950 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1434/2881] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 01:39:02 2022] POST /identity/v3/users => generated 290 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 28 01:39:02.771100 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e0cf568-9750-4b07-8837-fe9fcdd9d7e2 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:39:02.782391 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e0cf568-9750-4b07-8837-fe9fcdd9d7e2 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:39:02.787393 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfc60934-e921-41f6-a704-5386d4ffbb72 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:39:02.787998 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfc60934-e921-41f6-a704-5386d4ffbb72 admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:39:02.788276 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfc60934-e921-41f6-a704-5386d4ffbb72 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:39:02.788540 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfc60934-e921-41f6-a704-5386d4ffbb72 admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:39:02.789215 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfc60934-e921-41f6-a704-5386d4ffbb72 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:39:02.790976 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfc60934-e921-41f6-a704-5386d4ffbb72 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:39:02.801703 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1448/2882] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 01:39:02 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 01:39:02.806310 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-329b51d0-f488-4b2c-9e1e-cbcd7dd5d48e admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:39:02.816472 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-329b51d0-f488-4b2c-9e1e-cbcd7dd5d48e admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:39:02.821374 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1f75527-51ab-40cc-8b59-14508ad80beb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:39:02.822028 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1f75527-51ab-40cc-8b59-14508ad80beb admin admin] REQUEST_METHOD: `PUT` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:39:02.822335 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1f75527-51ab-40cc-8b59-14508ad80beb admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:39:02.822605 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1f75527-51ab-40cc-8b59-14508ad80beb admin admin] PATH_INFO: `/v3/projects/a2b56b1dd26443deb6aebea90cc7014f/users/561f824505c64eebaddc8482b1264851/roles/f08d6043261747e3b4763c744eef7dfb` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:39:02.838762 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1f75527-51ab-40cc-8b59-14508ad80beb admin admin] RBAC: Authorizing `identity:create_grant(project_id=a2b56b1dd26443deb6aebea90cc7014f, user_id=561f824505c64eebaddc8482b1264851, role_id=f08d6043261747e3b4763c744eef7dfb)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:39:02.841478 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1f75527-51ab-40cc-8b59-14508ad80beb admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:39:02.858895 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1435/2883] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 01:39:02 2022] PUT /identity/v3/projects/a2b56b1dd26443deb6aebea90cc7014f/users/561f824505c64eebaddc8482b1264851/roles/f08d6043261747e3b4763c744eef7dfb => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:39:02.863136 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfc60934-e921-41f6-a704-5386d4ffbb72 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:39:02.873520 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfc60934-e921-41f6-a704-5386d4ffbb72 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:39:02.895074 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:DELETE=6,keystone:SELECT=90,keystone:INSERT=4 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:39:02.944711 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49668436-f793-4b13-a91e-fbee8e530f10 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:39:02.945363 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49668436-f793-4b13-a91e-fbee8e530f10 admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:39:02.945638 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49668436-f793-4b13-a91e-fbee8e530f10 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:39:02.945902 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49668436-f793-4b13-a91e-fbee8e530f10 admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:39:02.946565 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49668436-f793-4b13-a91e-fbee8e530f10 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:39:02.948191 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49668436-f793-4b13-a91e-fbee8e530f10 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:39:02.958878 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1449/2884] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 01:39:02 2022] GET /identity/v3/roles => generated 1420 bytes in 96 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 01:39:02.964543 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1f75527-51ab-40cc-8b59-14508ad80beb admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:39:02.970333 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:39:02.976579 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1f75527-51ab-40cc-8b59-14508ad80beb admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:39:02.983269 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-706d457c-f0a3-45e0-ade9-1d332d036b51 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:39:02.983932 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-706d457c-f0a3-45e0-ade9-1d332d036b51 admin admin] REQUEST_METHOD: `PUT` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:39:02.984206 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-706d457c-f0a3-45e0-ade9-1d332d036b51 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:39:02.984488 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-706d457c-f0a3-45e0-ade9-1d332d036b51 admin admin] PATH_INFO: `/v3/projects/a2b56b1dd26443deb6aebea90cc7014f/users/561f824505c64eebaddc8482b1264851/roles/f08d6043261747e3b4763c744eef7dfb` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:39:02.986740 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-706d457c-f0a3-45e0-ade9-1d332d036b51 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a2b56b1dd26443deb6aebea90cc7014f, user_id=561f824505c64eebaddc8482b1264851, role_id=f08d6043261747e3b4763c744eef7dfb)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:39:02.989440 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-706d457c-f0a3-45e0-ade9-1d332d036b51 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:39:03.003832 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1436/2885] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 01:39:02 2022] PUT /identity/v3/projects/a2b56b1dd26443deb6aebea90cc7014f/users/561f824505c64eebaddc8482b1264851/roles/f08d6043261747e3b4763c744eef7dfb => generated 0 bytes in 40 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:39:03.016116 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-927a5787-8e4c-4e52-a978-5a8dff2e860d None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:39:03.016486 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-927a5787-8e4c-4e52-a978-5a8dff2e860d None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:39:03.016818 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-927a5787-8e4c-4e52-a978-5a8dff2e860d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:39:03.055273 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-927a5787-8e4c-4e52-a978-5a8dff2e860d None None] MFA Rules not processed for user `561f824505c64eebaddc8482b1264851`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:39:03.056499 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-927a5787-8e4c-4e52-a978-5a8dff2e860d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:39:03.129218 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1450/2886] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:39:03 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:39:03.140783 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-706d457c-f0a3-45e0-ade9-1d332d036b51 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:39:03.142766 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-706d457c-f0a3-45e0-ade9-1d332d036b51 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:39:03.203562 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-706d457c-f0a3-45e0-ade9-1d332d036b51 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:39:03.207938 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df532b45-f26b-49b2-90a0-3bf375676de4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:39:03.208543 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df532b45-f26b-49b2-90a0-3bf375676de4 service cinder] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:39:03.208818 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df532b45-f26b-49b2-90a0-3bf375676de4 service cinder] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:39:03.209114 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df532b45-f26b-49b2-90a0-3bf375676de4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:39:03.219513 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df532b45-f26b-49b2-90a0-3bf375676de4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:39:03.220975 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df532b45-f26b-49b2-90a0-3bf375676de4 service cinder] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:39:03.231183 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1437/2887] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:39:03 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:39:03.283693 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-927a5787-8e4c-4e52-a978-5a8dff2e860d None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:39:03.285663 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-927a5787-8e4c-4e52-a978-5a8dff2e860d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:39:03.302937 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-927a5787-8e4c-4e52-a978-5a8dff2e860d None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:39:03.309211 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d67ba6df-45ab-4194-8406-1ee0575d3f37 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:39:03.310024 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d67ba6df-45ab-4194-8406-1ee0575d3f37 service cinder] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:39:03.310355 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d67ba6df-45ab-4194-8406-1ee0575d3f37 service cinder] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:39:03.310693 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d67ba6df-45ab-4194-8406-1ee0575d3f37 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:39:03.371873 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d67ba6df-45ab-4194-8406-1ee0575d3f37 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:39:03.373351 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d67ba6df-45ab-4194-8406-1ee0575d3f37 service cinder] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:39:03.382783 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1451/2888] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:39:03 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:39:06.788508 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df532b45-f26b-49b2-90a0-3bf375676de4 service cinder] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:39:06.864921 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df532b45-f26b-49b2-90a0-3bf375676de4 service cinder] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:39:06.870172 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d0c3320-f6d0-442e-90b6-408950680164 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:39:06.870804 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d0c3320-f6d0-442e-90b6-408950680164 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:39:06.871082 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d0c3320-f6d0-442e-90b6-408950680164 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:39:06.871357 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d0c3320-f6d0-442e-90b6-408950680164 admin admin] PATH_INFO: `/v3/users/7b516f3b75084aed874fa74c7f5ec007` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:39:06.872524 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d0c3320-f6d0-442e-90b6-408950680164 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7b516f3b75084aed874fa74c7f5ec007)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:39:06.874536 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d0c3320-f6d0-442e-90b6-408950680164 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:39:06.928516 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-8d0c3320-f6d0-442e-90b6-408950680164 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7b516f3b75084aed874fa74c7f5ec007'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:39:06.946028 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-8d0c3320-f6d0-442e-90b6-408950680164 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7b516f3b75084aed874fa74c7f5ec007'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:39:06.957152 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-8d0c3320-f6d0-442e-90b6-408950680164 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7b516f3b75084aed874fa74c7f5ec007'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:39:06.964773 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-8d0c3320-f6d0-442e-90b6-408950680164 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7b516f3b75084aed874fa74c7f5ec007'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:39:06.965452 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-8d0c3320-f6d0-442e-90b6-408950680164 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7b516f3b75084aed874fa74c7f5ec007'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:39:06.971401 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1438/2889] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 01:39:06 2022] DELETE /identity/v3/users/7b516f3b75084aed874fa74c7f5ec007 => generated 0 bytes in 184 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:39:06.976028 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06a2017f-4ab0-415a-93af-3b6f1b4dfb49 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:39:06.976506 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06a2017f-4ab0-415a-93af-3b6f1b4dfb49 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:39:06.976621 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06a2017f-4ab0-415a-93af-3b6f1b4dfb49 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:39:06.996412 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-06a2017f-4ab0-415a-93af-3b6f1b4dfb49 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:39:06.997645 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-06a2017f-4ab0-415a-93af-3b6f1b4dfb49 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:39:07.045246 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:39:07.064894 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1452/2890] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:39:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:39:07.075803 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d0c3320-f6d0-442e-90b6-408950680164 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:39:07.077865 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-8d0c3320-f6d0-442e-90b6-408950680164 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:39:07.091881 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:39:07.145874 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d0c3320-f6d0-442e-90b6-408950680164 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:39:07.151912 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bbd430c-e81e-4128-84be-09c8d2c19501 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:39:07.152746 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bbd430c-e81e-4128-84be-09c8d2c19501 service neutron] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:39:07.153559 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bbd430c-e81e-4128-84be-09c8d2c19501 service neutron] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:39:07.153559 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bbd430c-e81e-4128-84be-09c8d2c19501 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:39:07.165780 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bbd430c-e81e-4128-84be-09c8d2c19501 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:39:07.167254 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bbd430c-e81e-4128-84be-09c8d2c19501 service neutron] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:39:07.177513 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1439/2891] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:39:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:39:07.474304 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06a2017f-4ab0-415a-93af-3b6f1b4dfb49 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:39:07.476034 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-06a2017f-4ab0-415a-93af-3b6f1b4dfb49 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:39:07.490840 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06a2017f-4ab0-415a-93af-3b6f1b4dfb49 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:39:07.496398 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab6b4a2f-22af-48d1-bc25-9512d0ec16f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:39:07.496949 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab6b4a2f-22af-48d1-bc25-9512d0ec16f0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:39:07.497176 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab6b4a2f-22af-48d1-bc25-9512d0ec16f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:39:07.497448 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab6b4a2f-22af-48d1-bc25-9512d0ec16f0 admin admin] PATH_INFO: `/v3/projects/83aa7ef80a3b4a259611f6f684644211` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:39:07.498725 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab6b4a2f-22af-48d1-bc25-9512d0ec16f0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=83aa7ef80a3b4a259611f6f684644211)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:39:07.500677 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab6b4a2f-22af-48d1-bc25-9512d0ec16f0 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:39:07.549478 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-ab6b4a2f-22af-48d1-bc25-9512d0ec16f0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '83aa7ef80a3b4a259611f6f684644211', 'request_id': 'req-ab6b4a2f-22af-48d1-bc25-9512d0ec16f0'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:39:07.556359 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-ab6b4a2f-22af-48d1-bc25-9512d0ec16f0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '83aa7ef80a3b4a259611f6f684644211', 'request_id': 'req-ab6b4a2f-22af-48d1-bc25-9512d0ec16f0'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:39:07.563326 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-ab6b4a2f-22af-48d1-bc25-9512d0ec16f0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '83aa7ef80a3b4a259611f6f684644211', 'request_id': 'req-ab6b4a2f-22af-48d1-bc25-9512d0ec16f0'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:39:07.580393 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-ab6b4a2f-22af-48d1-bc25-9512d0ec16f0 admin admin] The token cache is being invalidate because project 83aa7ef80a3b4a259611f6f684644211 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81508) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 01:39:07.580599 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-ab6b4a2f-22af-48d1-bc25-9512d0ec16f0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:39:07.581251 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-ab6b4a2f-22af-48d1-bc25-9512d0ec16f0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:39:07.582737 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1453/2892] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 01:39:07 2022] DELETE /identity/v3/projects/83aa7ef80a3b4a259611f6f684644211 => generated 0 bytes in 109 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 28 01:39:07.587019 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bbd430c-e81e-4128-84be-09c8d2c19501 service neutron] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:39:07.597910 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bbd430c-e81e-4128-84be-09c8d2c19501 service neutron] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:39:07.657998 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b9b9ad6-9fa9-4314-8436-8bffed8c9b88 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:39:07.658628 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b9b9ad6-9fa9-4314-8436-8bffed8c9b88 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:39:07.658905 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b9b9ad6-9fa9-4314-8436-8bffed8c9b88 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:39:07.659173 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b9b9ad6-9fa9-4314-8436-8bffed8c9b88 admin admin] PATH_INFO: `/v3/users/561f824505c64eebaddc8482b1264851` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:39:07.660333 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b9b9ad6-9fa9-4314-8436-8bffed8c9b88 admin admin] RBAC: Authorizing `identity:delete_user(user_id=561f824505c64eebaddc8482b1264851)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:39:07.662281 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b9b9ad6-9fa9-4314-8436-8bffed8c9b88 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:39:07.713365 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-9b9b9ad6-9fa9-4314-8436-8bffed8c9b88 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '561f824505c64eebaddc8482b1264851'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:39:07.726855 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-9b9b9ad6-9fa9-4314-8436-8bffed8c9b88 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '561f824505c64eebaddc8482b1264851'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:39:07.736170 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-9b9b9ad6-9fa9-4314-8436-8bffed8c9b88 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '561f824505c64eebaddc8482b1264851'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:39:07.743479 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-9b9b9ad6-9fa9-4314-8436-8bffed8c9b88 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '561f824505c64eebaddc8482b1264851'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:39:07.744150 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-9b9b9ad6-9fa9-4314-8436-8bffed8c9b88 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '561f824505c64eebaddc8482b1264851'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:39:07.750431 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1440/2893] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 01:39:07 2022] DELETE /identity/v3/users/561f824505c64eebaddc8482b1264851 => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:39:08.044634 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab6b4a2f-22af-48d1-bc25-9512d0ec16f0 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:39:08.046238 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-ab6b4a2f-22af-48d1-bc25-9512d0ec16f0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:39:08.059825 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab6b4a2f-22af-48d1-bc25-9512d0ec16f0 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:39:08.129387 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d79a2386-6dc8-4d3f-a34c-69e7069816f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:39:08.130059 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d79a2386-6dc8-4d3f-a34c-69e7069816f1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:39:08.130361 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d79a2386-6dc8-4d3f-a34c-69e7069816f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:39:08.131148 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d79a2386-6dc8-4d3f-a34c-69e7069816f1 admin admin] PATH_INFO: `/v3/projects/a2b56b1dd26443deb6aebea90cc7014f` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:39:08.132571 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d79a2386-6dc8-4d3f-a34c-69e7069816f1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a2b56b1dd26443deb6aebea90cc7014f)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:39:08.134819 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d79a2386-6dc8-4d3f-a34c-69e7069816f1 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:39:08.189244 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-d79a2386-6dc8-4d3f-a34c-69e7069816f1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a2b56b1dd26443deb6aebea90cc7014f', 'request_id': 'req-d79a2386-6dc8-4d3f-a34c-69e7069816f1'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:39:08.195875 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-d79a2386-6dc8-4d3f-a34c-69e7069816f1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a2b56b1dd26443deb6aebea90cc7014f', 'request_id': 'req-d79a2386-6dc8-4d3f-a34c-69e7069816f1'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:39:08.202419 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-d79a2386-6dc8-4d3f-a34c-69e7069816f1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a2b56b1dd26443deb6aebea90cc7014f', 'request_id': 'req-d79a2386-6dc8-4d3f-a34c-69e7069816f1'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:39:08.219155 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:39:08.225714 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-d79a2386-6dc8-4d3f-a34c-69e7069816f1 admin admin] The token cache is being invalidate because project a2b56b1dd26443deb6aebea90cc7014f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81508) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 01:39:08.225831 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-d79a2386-6dc8-4d3f-a34c-69e7069816f1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:39:08.226628 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-d79a2386-6dc8-4d3f-a34c-69e7069816f1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:39:08.228922 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1454/2894] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 01:39:08 2022] DELETE /identity/v3/projects/a2b56b1dd26443deb6aebea90cc7014f => generated 0 bytes in 185 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:39:08.233981 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12852b05-58d8-4f65-b41d-636df4c9970f None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:39:08.234529 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12852b05-58d8-4f65-b41d-636df4c9970f None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:39:08.234807 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12852b05-58d8-4f65-b41d-636df4c9970f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:39:08.255895 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-12852b05-58d8-4f65-b41d-636df4c9970f None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:39:08.257156 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-12852b05-58d8-4f65-b41d-636df4c9970f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:39:08.288409 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:39:08.338278 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1441/2895] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:39:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 28 01:39:08.346537 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85095623-1ffe-48b4-b838-3f09dbc4c351 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:39:08.346820 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85095623-1ffe-48b4-b838-3f09dbc4c351 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:39:08.347091 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85095623-1ffe-48b4-b838-3f09dbc4c351 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:39:08.370456 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-85095623-1ffe-48b4-b838-3f09dbc4c351 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:39:08.376680 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-85095623-1ffe-48b4-b838-3f09dbc4c351 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:39:08.388005 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1455/2896] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:39:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 43 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:39:08.393315 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12852b05-58d8-4f65-b41d-636df4c9970f None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:39:08.404702 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12852b05-58d8-4f65-b41d-636df4c9970f None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:39:08.410876 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a04137d8-529c-4451-9699-7576e040e022 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:39:08.411569 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a04137d8-529c-4451-9699-7576e040e022 admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:39:08.411840 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a04137d8-529c-4451-9699-7576e040e022 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:39:08.412108 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a04137d8-529c-4451-9699-7576e040e022 admin admin] PATH_INFO: `/v3/domains` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:39:08.412763 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a04137d8-529c-4451-9699-7576e040e022 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 01:39:08.413082 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a04137d8-529c-4451-9699-7576e040e022 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:39:08.414725 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a04137d8-529c-4451-9699-7576e040e022 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:39:08.429031 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1442/2897] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Aug 28 01:39:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 01:39:08.434035 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85095623-1ffe-48b4-b838-3f09dbc4c351 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:39:08.444922 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85095623-1ffe-48b4-b838-3f09dbc4c351 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:39:08.450085 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0eb83da-0e9b-4f3a-a78f-ebfdd624188b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:39:08.450706 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0eb83da-0e9b-4f3a-a78f-ebfdd624188b admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:39:08.450983 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0eb83da-0e9b-4f3a-a78f-ebfdd624188b admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:39:08.451244 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0eb83da-0e9b-4f3a-a78f-ebfdd624188b admin admin] PATH_INFO: `/v3/projects` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:39:08.451977 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0eb83da-0e9b-4f3a-a78f-ebfdd624188b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:39:08.453963 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0eb83da-0e9b-4f3a-a78f-ebfdd624188b admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:39:08.484086 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1456/2898] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 01:39:08 2022] POST /identity/v3/projects => generated 385 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 01:39:08.489061 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a04137d8-529c-4451-9699-7576e040e022 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:39:08.500128 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a04137d8-529c-4451-9699-7576e040e022 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:39:08.565429 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6ba1701-f642-493d-a4bd-38cc33a944ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:39:08.566083 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6ba1701-f642-493d-a4bd-38cc33a944ea admin admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:39:08.566365 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6ba1701-f642-493d-a4bd-38cc33a944ea admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:39:08.566637 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6ba1701-f642-493d-a4bd-38cc33a944ea admin admin] PATH_INFO: `/v3/users` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:39:08.567288 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6ba1701-f642-493d-a4bd-38cc33a944ea admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:39:08.569323 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6ba1701-f642-493d-a4bd-38cc33a944ea admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:39:08.597008 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1443/2899] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 01:39:08 2022] POST /identity/v3/users => generated 293 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 28 01:39:08.601030 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0eb83da-0e9b-4f3a-a78f-ebfdd624188b admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:39:08.612187 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0eb83da-0e9b-4f3a-a78f-ebfdd624188b admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:39:08.617488 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca0caf64-c342-495f-b9a7-5688900ab171 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:39:08.618128 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca0caf64-c342-495f-b9a7-5688900ab171 admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:39:08.618411 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca0caf64-c342-495f-b9a7-5688900ab171 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:39:08.618672 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca0caf64-c342-495f-b9a7-5688900ab171 admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:39:08.619324 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca0caf64-c342-495f-b9a7-5688900ab171 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:39:08.621103 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca0caf64-c342-495f-b9a7-5688900ab171 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:39:08.631657 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1457/2900] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 01:39:08 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 01:39:08.636883 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6ba1701-f642-493d-a4bd-38cc33a944ea admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:39:08.649197 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6ba1701-f642-493d-a4bd-38cc33a944ea admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:39:08.654189 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63445ee6-6843-441b-8086-29114b9e2603 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:39:08.654999 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63445ee6-6843-441b-8086-29114b9e2603 admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:39:08.655287 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63445ee6-6843-441b-8086-29114b9e2603 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:39:08.655597 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63445ee6-6843-441b-8086-29114b9e2603 admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:39:08.656461 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63445ee6-6843-441b-8086-29114b9e2603 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:39:08.658442 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63445ee6-6843-441b-8086-29114b9e2603 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:39:08.671282 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1444/2901] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 01:39:08 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 01:39:08.676109 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca0caf64-c342-495f-b9a7-5688900ab171 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:39:08.688556 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca0caf64-c342-495f-b9a7-5688900ab171 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:39:08.694743 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56433e7e-08b1-43ca-9e66-b6dcb483f1eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:39:08.695431 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56433e7e-08b1-43ca-9e66-b6dcb483f1eb admin admin] REQUEST_METHOD: `PUT` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:39:08.695695 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56433e7e-08b1-43ca-9e66-b6dcb483f1eb admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:39:08.695972 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56433e7e-08b1-43ca-9e66-b6dcb483f1eb admin admin] PATH_INFO: `/v3/projects/2a448de62ac34ec3a81a223fb507708b/users/20a0628e606648a4a48608c3a520fe1a/roles/0487ef7e943e4701915cb8ce0f2899d7` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:39:08.712448 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56433e7e-08b1-43ca-9e66-b6dcb483f1eb admin admin] RBAC: Authorizing `identity:create_grant(project_id=2a448de62ac34ec3a81a223fb507708b, user_id=20a0628e606648a4a48608c3a520fe1a, role_id=0487ef7e943e4701915cb8ce0f2899d7)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:39:08.715177 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56433e7e-08b1-43ca-9e66-b6dcb483f1eb admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:39:08.732602 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1458/2902] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 01:39:08 2022] PUT /identity/v3/projects/2a448de62ac34ec3a81a223fb507708b/users/20a0628e606648a4a48608c3a520fe1a/roles/0487ef7e943e4701915cb8ce0f2899d7 => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:39:08.744601 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4b50a8c-2399-4ae8-815e-6293a70dcb2c None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:39:08.744865 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4b50a8c-2399-4ae8-815e-6293a70dcb2c None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:39:08.745193 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4b50a8c-2399-4ae8-815e-6293a70dcb2c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:39:08.774425 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-a4b50a8c-2399-4ae8-815e-6293a70dcb2c None None] MFA Rules not processed for user `20a0628e606648a4a48608c3a520fe1a`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:39:08.775666 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-a4b50a8c-2399-4ae8-815e-6293a70dcb2c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:39:08.852182 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1445/2903] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:39:08 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:39:08.864253 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56433e7e-08b1-43ca-9e66-b6dcb483f1eb admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:39:08.866702 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-56433e7e-08b1-43ca-9e66-b6dcb483f1eb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:39:08.937946 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56433e7e-08b1-43ca-9e66-b6dcb483f1eb admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:39:08.941948 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88993ffe-8d12-4c2c-b753-1cbc29e7980a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:39:08.942561 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88993ffe-8d12-4c2c-b753-1cbc29e7980a service cinder] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:39:08.942839 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88993ffe-8d12-4c2c-b753-1cbc29e7980a service cinder] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:39:08.943115 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88993ffe-8d12-4c2c-b753-1cbc29e7980a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:39:08.952107 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88993ffe-8d12-4c2c-b753-1cbc29e7980a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:39:08.953574 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88993ffe-8d12-4c2c-b753-1cbc29e7980a service cinder] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:39:08.962709 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1459/2904] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:39:08 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:39:08.983555 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4b50a8c-2399-4ae8-815e-6293a70dcb2c None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:39:09.015789 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:39:09.052346 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4b50a8c-2399-4ae8-815e-6293a70dcb2c None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:39:09.057601 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5686e20-3295-4af2-a436-7d0e69a85f1f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:39:09.058348 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5686e20-3295-4af2-a436-7d0e69a85f1f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:39:09.058502 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5686e20-3295-4af2-a436-7d0e69a85f1f admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:39:09.058788 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5686e20-3295-4af2-a436-7d0e69a85f1f admin admin] PATH_INFO: `/v3/users/20a0628e606648a4a48608c3a520fe1a` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:39:09.059925 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5686e20-3295-4af2-a436-7d0e69a85f1f admin admin] RBAC: Authorizing `identity:delete_user(user_id=20a0628e606648a4a48608c3a520fe1a)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:39:09.061926 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5686e20-3295-4af2-a436-7d0e69a85f1f admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:39:09.118230 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-e5686e20-3295-4af2-a436-7d0e69a85f1f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '20a0628e606648a4a48608c3a520fe1a'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:39:09.135024 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-e5686e20-3295-4af2-a436-7d0e69a85f1f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '20a0628e606648a4a48608c3a520fe1a'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:39:09.146608 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-e5686e20-3295-4af2-a436-7d0e69a85f1f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '20a0628e606648a4a48608c3a520fe1a'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:39:09.153889 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-e5686e20-3295-4af2-a436-7d0e69a85f1f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '20a0628e606648a4a48608c3a520fe1a'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:39:09.154561 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-e5686e20-3295-4af2-a436-7d0e69a85f1f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '20a0628e606648a4a48608c3a520fe1a'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:39:09.161499 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1446/2905] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 01:39:08 2022] DELETE /identity/v3/users/20a0628e606648a4a48608c3a520fe1a => generated 0 bytes in 179 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:39:09.167294 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cdc3c1d-f160-4510-af8e-50ea240f1688 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:39:09.168472 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cdc3c1d-f160-4510-af8e-50ea240f1688 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:39:09.168472 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cdc3c1d-f160-4510-af8e-50ea240f1688 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:39:09.190741 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-4cdc3c1d-f160-4510-af8e-50ea240f1688 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:39:09.191861 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-4cdc3c1d-f160-4510-af8e-50ea240f1688 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:39:09.264177 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1460/2906] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:39:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 28 01:39:09.274790 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5686e20-3295-4af2-a436-7d0e69a85f1f admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:39:09.277733 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-e5686e20-3295-4af2-a436-7d0e69a85f1f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:39:09.345603 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5686e20-3295-4af2-a436-7d0e69a85f1f admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:39:09.350201 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35add157-b1e7-47e0-aa93-a44165cd9cea service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:39:09.350818 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35add157-b1e7-47e0-aa93-a44165cd9cea service neutron] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:39:09.351081 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35add157-b1e7-47e0-aa93-a44165cd9cea service neutron] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:39:09.351352 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35add157-b1e7-47e0-aa93-a44165cd9cea service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:39:09.361289 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35add157-b1e7-47e0-aa93-a44165cd9cea service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:39:09.362739 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35add157-b1e7-47e0-aa93-a44165cd9cea service neutron] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:39:09.372249 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1447/2907] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:39:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 28 01:39:09.673973 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cdc3c1d-f160-4510-af8e-50ea240f1688 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:39:09.675451 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-4cdc3c1d-f160-4510-af8e-50ea240f1688 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:39:09.688468 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cdc3c1d-f160-4510-af8e-50ea240f1688 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:39:09.693839 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d41cd69f-aea1-472b-a755-3a5b4895f204 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:39:09.694426 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d41cd69f-aea1-472b-a755-3a5b4895f204 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:39:09.694704 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d41cd69f-aea1-472b-a755-3a5b4895f204 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:39:09.694978 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d41cd69f-aea1-472b-a755-3a5b4895f204 admin admin] PATH_INFO: `/v3/projects/2a448de62ac34ec3a81a223fb507708b` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:39:09.696226 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d41cd69f-aea1-472b-a755-3a5b4895f204 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2a448de62ac34ec3a81a223fb507708b)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:39:09.698257 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d41cd69f-aea1-472b-a755-3a5b4895f204 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:39:09.750988 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=96,keystone:INSERT=2,keystone:DELETE=2 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:39:09.769099 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-d41cd69f-aea1-472b-a755-3a5b4895f204 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2a448de62ac34ec3a81a223fb507708b', 'request_id': 'req-d41cd69f-aea1-472b-a755-3a5b4895f204'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:39:09.775041 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-d41cd69f-aea1-472b-a755-3a5b4895f204 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2a448de62ac34ec3a81a223fb507708b', 'request_id': 'req-d41cd69f-aea1-472b-a755-3a5b4895f204'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:39:09.781556 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-d41cd69f-aea1-472b-a755-3a5b4895f204 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2a448de62ac34ec3a81a223fb507708b', 'request_id': 'req-d41cd69f-aea1-472b-a755-3a5b4895f204'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:39:09.796675 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-d41cd69f-aea1-472b-a755-3a5b4895f204 admin admin] The token cache is being invalidate because project 2a448de62ac34ec3a81a223fb507708b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81508) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 01:39:09.796997 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-d41cd69f-aea1-472b-a755-3a5b4895f204 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:39:09.797705 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-d41cd69f-aea1-472b-a755-3a5b4895f204 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:39:09.799227 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1461/2908] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 01:39:09 2022] DELETE /identity/v3/projects/2a448de62ac34ec3a81a223fb507708b => generated 0 bytes in 126 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:39:09.805790 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-595395bb-a404-4a49-933a-ee2cad19527b None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:39:09.806105 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-595395bb-a404-4a49-933a-ee2cad19527b None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:39:09.806432 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-595395bb-a404-4a49-933a-ee2cad19527b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:39:09.831045 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-595395bb-a404-4a49-933a-ee2cad19527b None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:39:09.832606 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-595395bb-a404-4a49-933a-ee2cad19527b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:39:09.883173 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:39:09.899416 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1448/2909] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:39:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:39:09.908828 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c4f3b71-123c-4d58-91af-3937e8295765 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:39:09.909142 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c4f3b71-123c-4d58-91af-3937e8295765 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:39:09.909450 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c4f3b71-123c-4d58-91af-3937e8295765 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:39:09.930173 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-0c4f3b71-123c-4d58-91af-3937e8295765 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:39:09.931397 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-0c4f3b71-123c-4d58-91af-3937e8295765 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:39:09.940744 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1462/2910] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:39:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 28 01:39:09.945570 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-595395bb-a404-4a49-933a-ee2cad19527b None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:39:09.956058 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-595395bb-a404-4a49-933a-ee2cad19527b None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:39:09.961799 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86b255a4-023b-42db-a02b-f366e581c428 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:39:09.962401 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86b255a4-023b-42db-a02b-f366e581c428 admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:39:09.962677 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86b255a4-023b-42db-a02b-f366e581c428 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:39:09.962930 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86b255a4-023b-42db-a02b-f366e581c428 admin admin] PATH_INFO: `/v3/domains` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:39:09.963584 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86b255a4-023b-42db-a02b-f366e581c428 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81507) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 01:39:09.963885 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86b255a4-023b-42db-a02b-f366e581c428 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:39:09.965528 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86b255a4-023b-42db-a02b-f366e581c428 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:39:09.977680 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1449/2911] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Aug 28 01:39:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 01:39:09.982418 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c4f3b71-123c-4d58-91af-3937e8295765 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:39:09.991817 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c4f3b71-123c-4d58-91af-3937e8295765 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:39:09.996896 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a68e5af9-1f09-4111-8320-92158d1df13d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:39:09.997519 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a68e5af9-1f09-4111-8320-92158d1df13d admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:39:09.997789 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a68e5af9-1f09-4111-8320-92158d1df13d admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:39:09.998053 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a68e5af9-1f09-4111-8320-92158d1df13d admin admin] PATH_INFO: `/v3/projects` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:39:09.998672 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a68e5af9-1f09-4111-8320-92158d1df13d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:39:10.000599 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a68e5af9-1f09-4111-8320-92158d1df13d admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:39:10.032802 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1463/2912] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 01:39:09 2022] POST /identity/v3/projects => generated 383 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 01:39:10.038978 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86b255a4-023b-42db-a02b-f366e581c428 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:39:10.049901 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86b255a4-023b-42db-a02b-f366e581c428 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:39:10.115497 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d6f0b13-8d58-4971-a63c-7f8cef0504de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:39:10.116121 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d6f0b13-8d58-4971-a63c-7f8cef0504de admin admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:39:10.116370 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d6f0b13-8d58-4971-a63c-7f8cef0504de admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:39:10.116631 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d6f0b13-8d58-4971-a63c-7f8cef0504de admin admin] PATH_INFO: `/v3/users` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:39:10.117353 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d6f0b13-8d58-4971-a63c-7f8cef0504de admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:39:10.119433 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d6f0b13-8d58-4971-a63c-7f8cef0504de admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:39:10.143490 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1450/2913] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 01:39:10 2022] POST /identity/v3/users => generated 292 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 01:39:10.148266 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a68e5af9-1f09-4111-8320-92158d1df13d admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:39:10.158805 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a68e5af9-1f09-4111-8320-92158d1df13d admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:39:10.163546 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d440e1b-cb7c-461a-adf0-3a9b87c90f16 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:39:10.164152 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d440e1b-cb7c-461a-adf0-3a9b87c90f16 admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:39:10.164431 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d440e1b-cb7c-461a-adf0-3a9b87c90f16 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:39:10.164701 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d440e1b-cb7c-461a-adf0-3a9b87c90f16 admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:39:10.165387 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d440e1b-cb7c-461a-adf0-3a9b87c90f16 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:39:10.167034 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d440e1b-cb7c-461a-adf0-3a9b87c90f16 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:39:10.182892 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1464/2914] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 01:39:10 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 01:39:10.188486 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d6f0b13-8d58-4971-a63c-7f8cef0504de admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:39:10.199608 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d6f0b13-8d58-4971-a63c-7f8cef0504de admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:39:10.205662 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1df8c3f4-a642-4e3b-8850-4020b371bc6a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:39:10.206246 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1df8c3f4-a642-4e3b-8850-4020b371bc6a admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:39:10.206514 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1df8c3f4-a642-4e3b-8850-4020b371bc6a admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:39:10.206784 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1df8c3f4-a642-4e3b-8850-4020b371bc6a admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:39:10.207439 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1df8c3f4-a642-4e3b-8850-4020b371bc6a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:39:10.209064 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1df8c3f4-a642-4e3b-8850-4020b371bc6a admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:39:10.223751 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1451/2915] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 01:39:10 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 01:39:10.228426 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d440e1b-cb7c-461a-adf0-3a9b87c90f16 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:39:10.238978 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d440e1b-cb7c-461a-adf0-3a9b87c90f16 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:39:10.244668 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afd6ff9f-9cf1-45f5-b570-9e0e3156e4a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:39:10.245389 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd6ff9f-9cf1-45f5-b570-9e0e3156e4a3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:39:10.245663 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd6ff9f-9cf1-45f5-b570-9e0e3156e4a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:39:10.245931 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd6ff9f-9cf1-45f5-b570-9e0e3156e4a3 admin admin] PATH_INFO: `/v3/projects/5c3daf077901407095e8648c6032d1b2/users/0d9202caf53f4d9e86d1f2b2a0719004/roles/0487ef7e943e4701915cb8ce0f2899d7` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:39:10.260896 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afd6ff9f-9cf1-45f5-b570-9e0e3156e4a3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5c3daf077901407095e8648c6032d1b2, user_id=0d9202caf53f4d9e86d1f2b2a0719004, role_id=0487ef7e943e4701915cb8ce0f2899d7)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:39:10.263557 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afd6ff9f-9cf1-45f5-b570-9e0e3156e4a3 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:39:10.279083 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1465/2916] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 01:39:10 2022] PUT /identity/v3/projects/5c3daf077901407095e8648c6032d1b2/users/0d9202caf53f4d9e86d1f2b2a0719004/roles/0487ef7e943e4701915cb8ce0f2899d7 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:39:10.289804 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5aa9fee-8108-4635-998c-6325671e215d None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:39:10.290373 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5aa9fee-8108-4635-998c-6325671e215d None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:39:10.290373 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5aa9fee-8108-4635-998c-6325671e215d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:39:10.321185 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-f5aa9fee-8108-4635-998c-6325671e215d None None] MFA Rules not processed for user `0d9202caf53f4d9e86d1f2b2a0719004`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:39:10.322421 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-f5aa9fee-8108-4635-998c-6325671e215d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:39:10.395633 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1452/2917] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:39:10 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:39:10.400638 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afd6ff9f-9cf1-45f5-b570-9e0e3156e4a3 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:39:10.462674 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afd6ff9f-9cf1-45f5-b570-9e0e3156e4a3 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:39:10.468482 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efe65047-0dc2-474d-806a-e3888740b87d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:39:10.469127 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efe65047-0dc2-474d-806a-e3888740b87d admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:39:10.469403 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efe65047-0dc2-474d-806a-e3888740b87d admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:39:10.469672 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efe65047-0dc2-474d-806a-e3888740b87d admin admin] PATH_INFO: `/v3/projects` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:39:10.470305 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efe65047-0dc2-474d-806a-e3888740b87d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:39:10.472256 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efe65047-0dc2-474d-806a-e3888740b87d admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:39:10.506126 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1466/2918] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 01:39:10 2022] POST /identity/v3/projects => generated 385 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 01:39:10.510598 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5aa9fee-8108-4635-998c-6325671e215d None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:39:10.523525 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5aa9fee-8108-4635-998c-6325671e215d None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:39:10.571137 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:39:10.586550 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ac11597-c69e-4b44-9637-37eeea1179fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:39:10.587177 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ac11597-c69e-4b44-9637-37eeea1179fe admin admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:39:10.587452 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ac11597-c69e-4b44-9637-37eeea1179fe admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:39:10.587715 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ac11597-c69e-4b44-9637-37eeea1179fe admin admin] PATH_INFO: `/v3/users` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:39:10.588347 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ac11597-c69e-4b44-9637-37eeea1179fe admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:39:10.590296 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ac11597-c69e-4b44-9637-37eeea1179fe admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:39:10.615001 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1453/2919] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 01:39:10 2022] POST /identity/v3/users => generated 293 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 01:39:10.619206 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efe65047-0dc2-474d-806a-e3888740b87d admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:39:10.631243 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efe65047-0dc2-474d-806a-e3888740b87d admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:39:10.636807 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27ba2e8e-5378-48fe-befb-b4a810eaccc1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:39:10.637454 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27ba2e8e-5378-48fe-befb-b4a810eaccc1 admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:39:10.637728 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27ba2e8e-5378-48fe-befb-b4a810eaccc1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:39:10.637991 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27ba2e8e-5378-48fe-befb-b4a810eaccc1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:39:10.638638 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27ba2e8e-5378-48fe-befb-b4a810eaccc1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:39:10.640391 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27ba2e8e-5378-48fe-befb-b4a810eaccc1 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:39:10.654965 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1467/2920] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 01:39:10 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 01:39:10.659176 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ac11597-c69e-4b44-9637-37eeea1179fe admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:39:10.669914 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ac11597-c69e-4b44-9637-37eeea1179fe admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:39:10.675389 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b8e6d43-6b36-4894-a8ea-0e20bf2b8613 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:39:10.675988 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b8e6d43-6b36-4894-a8ea-0e20bf2b8613 admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:39:10.676264 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b8e6d43-6b36-4894-a8ea-0e20bf2b8613 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:39:10.676605 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b8e6d43-6b36-4894-a8ea-0e20bf2b8613 admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:39:10.677352 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b8e6d43-6b36-4894-a8ea-0e20bf2b8613 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:39:10.679622 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b8e6d43-6b36-4894-a8ea-0e20bf2b8613 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:39:10.692053 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1454/2921] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 01:39:10 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 01:39:10.696215 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27ba2e8e-5378-48fe-befb-b4a810eaccc1 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:39:10.706400 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27ba2e8e-5378-48fe-befb-b4a810eaccc1 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:39:10.711065 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27408913-b3dd-4781-b6fd-e74ede50c8ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:39:10.711706 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27408913-b3dd-4781-b6fd-e74ede50c8ca admin admin] REQUEST_METHOD: `PUT` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:39:10.711979 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27408913-b3dd-4781-b6fd-e74ede50c8ca admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:39:10.712246 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27408913-b3dd-4781-b6fd-e74ede50c8ca admin admin] PATH_INFO: `/v3/projects/d836d94222404a828860b25626d5b94f/users/682138b0839b4e35b1b46b25d4dfed05/roles/0487ef7e943e4701915cb8ce0f2899d7` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:39:10.732751 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27408913-b3dd-4781-b6fd-e74ede50c8ca admin admin] RBAC: Authorizing `identity:create_grant(project_id=d836d94222404a828860b25626d5b94f, user_id=682138b0839b4e35b1b46b25d4dfed05, role_id=0487ef7e943e4701915cb8ce0f2899d7)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:39:10.735748 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27408913-b3dd-4781-b6fd-e74ede50c8ca admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:39:10.752284 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1468/2922] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 01:39:10 2022] PUT /identity/v3/projects/d836d94222404a828860b25626d5b94f/users/682138b0839b4e35b1b46b25d4dfed05/roles/0487ef7e943e4701915cb8ce0f2899d7 => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:39:10.765001 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea0f66dc-9a5f-4cf7-9ace-05b1accf7196 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:39:10.765396 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea0f66dc-9a5f-4cf7-9ace-05b1accf7196 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:39:10.765681 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea0f66dc-9a5f-4cf7-9ace-05b1accf7196 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:39:10.800010 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-ea0f66dc-9a5f-4cf7-9ace-05b1accf7196 None None] MFA Rules not processed for user `682138b0839b4e35b1b46b25d4dfed05`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:39:10.801294 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-ea0f66dc-9a5f-4cf7-9ace-05b1accf7196 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:39:10.875991 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1455/2923] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:39:10 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:39:10.880730 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27408913-b3dd-4781-b6fd-e74ede50c8ca admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:39:10.890858 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=90,keystone:DELETE=5,keystone:INSERT=5 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:39:10.955910 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27408913-b3dd-4781-b6fd-e74ede50c8ca admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:39:10.963105 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c82c89b-77c3-47d8-9e6a-199c0055e045 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:39:10.964051 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c82c89b-77c3-47d8-9e6a-199c0055e045 admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:39:10.964436 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c82c89b-77c3-47d8-9e6a-199c0055e045 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:39:10.964791 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c82c89b-77c3-47d8-9e6a-199c0055e045 admin admin] PATH_INFO: `/v3/projects` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:39:10.965723 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c82c89b-77c3-47d8-9e6a-199c0055e045 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:39:10.968401 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c82c89b-77c3-47d8-9e6a-199c0055e045 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:39:10.997826 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1469/2924] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 01:39:10 2022] POST /identity/v3/projects => generated 379 bytes in 118 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 01:39:11.002263 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea0f66dc-9a5f-4cf7-9ace-05b1accf7196 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:39:11.014411 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea0f66dc-9a5f-4cf7-9ace-05b1accf7196 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:39:11.082841 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb894bd4-659c-4c10-b108-ffccbdfbbca3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:39:11.083527 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb894bd4-659c-4c10-b108-ffccbdfbbca3 admin admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:39:11.083809 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb894bd4-659c-4c10-b108-ffccbdfbbca3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:39:11.084084 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb894bd4-659c-4c10-b108-ffccbdfbbca3 admin admin] PATH_INFO: `/v3/users` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:39:11.084716 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb894bd4-659c-4c10-b108-ffccbdfbbca3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:39:11.086773 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb894bd4-659c-4c10-b108-ffccbdfbbca3 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:39:11.109169 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1456/2925] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 01:39:11 2022] POST /identity/v3/users => generated 296 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 01:39:11.114566 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c82c89b-77c3-47d8-9e6a-199c0055e045 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:39:11.128672 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c82c89b-77c3-47d8-9e6a-199c0055e045 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:39:11.134229 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2969e85a-f824-4cdc-89d2-f3ce4602c623 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:39:11.134840 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2969e85a-f824-4cdc-89d2-f3ce4602c623 admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:39:11.135118 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2969e85a-f824-4cdc-89d2-f3ce4602c623 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:39:11.135386 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2969e85a-f824-4cdc-89d2-f3ce4602c623 admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:39:11.136026 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2969e85a-f824-4cdc-89d2-f3ce4602c623 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:39:11.137664 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2969e85a-f824-4cdc-89d2-f3ce4602c623 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:39:11.149107 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1470/2926] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 01:39:11 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 01:39:11.154092 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb894bd4-659c-4c10-b108-ffccbdfbbca3 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:39:11.165423 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb894bd4-659c-4c10-b108-ffccbdfbbca3 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:39:11.171366 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2f395b8-ccba-4a45-9812-7b6ffbe0e25a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:39:11.172068 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2f395b8-ccba-4a45-9812-7b6ffbe0e25a admin admin] REQUEST_METHOD: `PUT` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:39:11.172384 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2f395b8-ccba-4a45-9812-7b6ffbe0e25a admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:39:11.172611 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2f395b8-ccba-4a45-9812-7b6ffbe0e25a admin admin] PATH_INFO: `/v3/projects/d9b1ad09c7ba48e0b004eceab7d81069/users/abc37bd535cc47da965107446f03097f/roles/f08d6043261747e3b4763c744eef7dfb` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:39:11.189510 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2f395b8-ccba-4a45-9812-7b6ffbe0e25a admin admin] RBAC: Authorizing `identity:create_grant(project_id=d9b1ad09c7ba48e0b004eceab7d81069, user_id=abc37bd535cc47da965107446f03097f, role_id=f08d6043261747e3b4763c744eef7dfb)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:39:11.192166 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2f395b8-ccba-4a45-9812-7b6ffbe0e25a admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:39:11.206436 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1457/2927] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 01:39:11 2022] PUT /identity/v3/projects/d9b1ad09c7ba48e0b004eceab7d81069/users/abc37bd535cc47da965107446f03097f/roles/f08d6043261747e3b4763c744eef7dfb => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:39:11.211399 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2969e85a-f824-4cdc-89d2-f3ce4602c623 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:39:11.222399 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2969e85a-f824-4cdc-89d2-f3ce4602c623 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:39:11.287728 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44eb35b5-169c-420d-8050-cefb90caee5b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:39:11.288252 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44eb35b5-169c-420d-8050-cefb90caee5b admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:39:11.288533 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44eb35b5-169c-420d-8050-cefb90caee5b admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:39:11.288806 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44eb35b5-169c-420d-8050-cefb90caee5b admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:39:11.289580 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44eb35b5-169c-420d-8050-cefb90caee5b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:39:11.291223 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44eb35b5-169c-420d-8050-cefb90caee5b admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:39:11.302293 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1471/2928] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 01:39:11 2022] GET /identity/v3/roles => generated 1420 bytes in 92 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 01:39:11.307174 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2f395b8-ccba-4a45-9812-7b6ffbe0e25a admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:39:11.318498 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2f395b8-ccba-4a45-9812-7b6ffbe0e25a admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:39:11.323929 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-694bf7a2-1a12-4508-a648-f1de07e4b0a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:39:11.324578 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-694bf7a2-1a12-4508-a648-f1de07e4b0a5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:39:11.324849 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-694bf7a2-1a12-4508-a648-f1de07e4b0a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:39:11.325144 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-694bf7a2-1a12-4508-a648-f1de07e4b0a5 admin admin] PATH_INFO: `/v3/projects/d9b1ad09c7ba48e0b004eceab7d81069/users/abc37bd535cc47da965107446f03097f/roles/f08d6043261747e3b4763c744eef7dfb` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:39:11.327334 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-694bf7a2-1a12-4508-a648-f1de07e4b0a5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d9b1ad09c7ba48e0b004eceab7d81069, user_id=abc37bd535cc47da965107446f03097f, role_id=f08d6043261747e3b4763c744eef7dfb)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:39:11.329991 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-694bf7a2-1a12-4508-a648-f1de07e4b0a5 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:39:11.340440 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1458/2929] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 01:39:11 2022] PUT /identity/v3/projects/d9b1ad09c7ba48e0b004eceab7d81069/users/abc37bd535cc47da965107446f03097f/roles/f08d6043261747e3b4763c744eef7dfb => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:39:11.350746 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ecaf9a9-5b48-48b9-8bf6-884422167ff8 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:39:11.351031 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ecaf9a9-5b48-48b9-8bf6-884422167ff8 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:39:11.351305 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ecaf9a9-5b48-48b9-8bf6-884422167ff8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:39:11.378939 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-5ecaf9a9-5b48-48b9-8bf6-884422167ff8 None None] MFA Rules not processed for user `abc37bd535cc47da965107446f03097f`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:39:11.380157 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-5ecaf9a9-5b48-48b9-8bf6-884422167ff8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:39:11.451128 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1472/2930] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:39:11 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:39:11.463065 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-694bf7a2-1a12-4508-a648-f1de07e4b0a5 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:39:11.464828 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-694bf7a2-1a12-4508-a648-f1de07e4b0a5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:39:11.475246 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:39:11.523393 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-694bf7a2-1a12-4508-a648-f1de07e4b0a5 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:39:11.527095 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbe895ff-d815-4d32-a953-c97ce322d59c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:39:11.527697 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe895ff-d815-4d32-a953-c97ce322d59c service cinder] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:39:11.527975 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe895ff-d815-4d32-a953-c97ce322d59c service cinder] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:39:11.528238 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe895ff-d815-4d32-a953-c97ce322d59c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:39:11.588008 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbe895ff-d815-4d32-a953-c97ce322d59c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:39:11.589385 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbe895ff-d815-4d32-a953-c97ce322d59c service cinder] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:39:11.599389 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1459/2931] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:39:11 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 137 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:39:12.863400 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ecaf9a9-5b48-48b9-8bf6-884422167ff8 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:39:12.864988 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-5ecaf9a9-5b48-48b9-8bf6-884422167ff8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:39:12.871942 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:39:12.945155 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ecaf9a9-5b48-48b9-8bf6-884422167ff8 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:39:12.949914 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-081a3c1e-b3f7-4723-bff3-29215b33429c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:39:12.950564 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-081a3c1e-b3f7-4723-bff3-29215b33429c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:39:12.950845 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-081a3c1e-b3f7-4723-bff3-29215b33429c admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:39:12.951137 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-081a3c1e-b3f7-4723-bff3-29215b33429c admin admin] PATH_INFO: `/v3/users/2d813500dfa64549a505296aa2d8c16f` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:39:12.952395 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-081a3c1e-b3f7-4723-bff3-29215b33429c admin admin] RBAC: Authorizing `identity:delete_user(user_id=2d813500dfa64549a505296aa2d8c16f)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:39:12.954346 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-081a3c1e-b3f7-4723-bff3-29215b33429c admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:39:13.012487 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-081a3c1e-b3f7-4723-bff3-29215b33429c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2d813500dfa64549a505296aa2d8c16f'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:39:13.013567 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-081a3c1e-b3f7-4723-bff3-29215b33429c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2d813500dfa64549a505296aa2d8c16f'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:39:13.023854 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-081a3c1e-b3f7-4723-bff3-29215b33429c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2d813500dfa64549a505296aa2d8c16f'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:39:13.036357 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-081a3c1e-b3f7-4723-bff3-29215b33429c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2d813500dfa64549a505296aa2d8c16f'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:39:13.043070 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-081a3c1e-b3f7-4723-bff3-29215b33429c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2d813500dfa64549a505296aa2d8c16f'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:39:13.063731 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1473/2932] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 01:39:12 2022] DELETE /identity/v3/users/2d813500dfa64549a505296aa2d8c16f => generated 0 bytes in 201 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:39:13.068635 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e803e1b-2815-4a71-9a23-612ac68aaec5 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:39:13.069162 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e803e1b-2815-4a71-9a23-612ac68aaec5 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:39:13.069162 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e803e1b-2815-4a71-9a23-612ac68aaec5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:39:13.091284 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-5e803e1b-2815-4a71-9a23-612ac68aaec5 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:39:13.092504 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-5e803e1b-2815-4a71-9a23-612ac68aaec5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:39:13.152216 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-081a3c1e-b3f7-4723-bff3-29215b33429c admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:39:13.154009 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-081a3c1e-b3f7-4723-bff3-29215b33429c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:39:13.168542 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1460/2933] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:39:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Aug 28 01:39:13.180007 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e803e1b-2815-4a71-9a23-612ac68aaec5 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:39:13.181736 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-5e803e1b-2815-4a71-9a23-612ac68aaec5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:39:13.215598 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-081a3c1e-b3f7-4723-bff3-29215b33429c admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:39:13.219334 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36a87096-978e-48a0-a894-354aba724dd5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:39:13.219957 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36a87096-978e-48a0-a894-354aba724dd5 service cinder] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:39:13.220251 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36a87096-978e-48a0-a894-354aba724dd5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:39:13.220527 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36a87096-978e-48a0-a894-354aba724dd5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:39:13.222304 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-36a87096-978e-48a0-a894-354aba724dd5 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:39:13.244164 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e803e1b-2815-4a71-9a23-612ac68aaec5 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:39:13.248009 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f03932a6-2126-4f17-a9ca-167e0ffa8c7a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:39:13.248617 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f03932a6-2126-4f17-a9ca-167e0ffa8c7a service neutron] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:39:13.248893 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f03932a6-2126-4f17-a9ca-167e0ffa8c7a service neutron] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:39:13.249330 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f03932a6-2126-4f17-a9ca-167e0ffa8c7a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:39:13.257314 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:39:13.258917 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f03932a6-2126-4f17-a9ca-167e0ffa8c7a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:39:13.260626 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f03932a6-2126-4f17-a9ca-167e0ffa8c7a service neutron] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:39:13.270897 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1461/2934] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:39:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 28 01:39:13.283144 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36a87096-978e-48a0-a894-354aba724dd5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:39:13.284633 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36a87096-978e-48a0-a894-354aba724dd5 service cinder] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:39:13.293510 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1474/2935] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:39:13 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 142 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 28 01:39:13.564391 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f03932a6-2126-4f17-a9ca-167e0ffa8c7a service neutron] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:39:13.567150 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-f03932a6-2126-4f17-a9ca-167e0ffa8c7a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:39:13.595426 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=100 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:39:13.601361 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f03932a6-2126-4f17-a9ca-167e0ffa8c7a service neutron] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:39:13.611707 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a38c7ab-7d81-4f30-81f8-fa026b13c4c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:39:13.612397 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a38c7ab-7d81-4f30-81f8-fa026b13c4c1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:39:13.612777 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a38c7ab-7d81-4f30-81f8-fa026b13c4c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:39:13.613185 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a38c7ab-7d81-4f30-81f8-fa026b13c4c1 admin admin] PATH_INFO: `/v3/projects/eef563fbe4b64080bdb44afff3039930` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:39:13.614570 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a38c7ab-7d81-4f30-81f8-fa026b13c4c1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=eef563fbe4b64080bdb44afff3039930)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:39:13.616901 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a38c7ab-7d81-4f30-81f8-fa026b13c4c1 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:39:13.675754 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-1a38c7ab-7d81-4f30-81f8-fa026b13c4c1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'eef563fbe4b64080bdb44afff3039930', 'request_id': 'req-1a38c7ab-7d81-4f30-81f8-fa026b13c4c1'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:39:13.685334 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-1a38c7ab-7d81-4f30-81f8-fa026b13c4c1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'eef563fbe4b64080bdb44afff3039930', 'request_id': 'req-1a38c7ab-7d81-4f30-81f8-fa026b13c4c1'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:39:13.695391 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-1a38c7ab-7d81-4f30-81f8-fa026b13c4c1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'eef563fbe4b64080bdb44afff3039930', 'request_id': 'req-1a38c7ab-7d81-4f30-81f8-fa026b13c4c1'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:39:13.713254 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-1a38c7ab-7d81-4f30-81f8-fa026b13c4c1 admin admin] The token cache is being invalidate because project eef563fbe4b64080bdb44afff3039930 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81507) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 01:39:13.713708 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-1a38c7ab-7d81-4f30-81f8-fa026b13c4c1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:39:13.714679 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-1a38c7ab-7d81-4f30-81f8-fa026b13c4c1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:39:13.716479 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1462/2936] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 01:39:13 2022] DELETE /identity/v3/projects/eef563fbe4b64080bdb44afff3039930 => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:39:13.979625 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36a87096-978e-48a0-a894-354aba724dd5 service cinder] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:39:13.989351 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36a87096-978e-48a0-a894-354aba724dd5 service cinder] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:39:14.039875 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afbbf612-b7c9-4969-8230-e07d10ee8422 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:39:14.040486 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afbbf612-b7c9-4969-8230-e07d10ee8422 service cinder] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:39:14.040865 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afbbf612-b7c9-4969-8230-e07d10ee8422 service cinder] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:39:14.041030 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afbbf612-b7c9-4969-8230-e07d10ee8422 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:39:14.043081 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-afbbf612-b7c9-4969-8230-e07d10ee8422 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:39:14.127721 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afbbf612-b7c9-4969-8230-e07d10ee8422 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:39:14.129282 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afbbf612-b7c9-4969-8230-e07d10ee8422 service cinder] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:39:14.140532 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1475/2937] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:39:13 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 161 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:39:18.062207 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a38c7ab-7d81-4f30-81f8-fa026b13c4c1 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:39:18.063767 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-1a38c7ab-7d81-4f30-81f8-fa026b13c4c1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:39:18.150562 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a38c7ab-7d81-4f30-81f8-fa026b13c4c1 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:39:18.156127 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-898e5a11-7b07-415f-8fb7-60319056edfd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:39:18.156787 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-898e5a11-7b07-415f-8fb7-60319056edfd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:39:18.157213 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-898e5a11-7b07-415f-8fb7-60319056edfd admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:39:18.157490 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-898e5a11-7b07-415f-8fb7-60319056edfd admin admin] PATH_INFO: `/v3/users/0d9202caf53f4d9e86d1f2b2a0719004` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:39:18.158673 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-898e5a11-7b07-415f-8fb7-60319056edfd admin admin] RBAC: Authorizing `identity:delete_user(user_id=0d9202caf53f4d9e86d1f2b2a0719004)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:39:18.160610 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-898e5a11-7b07-415f-8fb7-60319056edfd admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:39:18.211334 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-898e5a11-7b07-415f-8fb7-60319056edfd admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0d9202caf53f4d9e86d1f2b2a0719004'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:39:18.223487 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-898e5a11-7b07-415f-8fb7-60319056edfd admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0d9202caf53f4d9e86d1f2b2a0719004'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:39:18.232282 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-898e5a11-7b07-415f-8fb7-60319056edfd admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0d9202caf53f4d9e86d1f2b2a0719004'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:39:18.239679 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-898e5a11-7b07-415f-8fb7-60319056edfd admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0d9202caf53f4d9e86d1f2b2a0719004'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:39:18.240345 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-898e5a11-7b07-415f-8fb7-60319056edfd admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0d9202caf53f4d9e86d1f2b2a0719004'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:39:18.246586 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1463/2938] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 01:39:18 2022] DELETE /identity/v3/users/0d9202caf53f4d9e86d1f2b2a0719004 => generated 0 bytes in 185 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:39:18.251864 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91a30eb9-03cc-41f1-8e8a-0d5df7c72358 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:39:18.252349 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91a30eb9-03cc-41f1-8e8a-0d5df7c72358 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:39:18.252470 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91a30eb9-03cc-41f1-8e8a-0d5df7c72358 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:39:18.273202 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-91a30eb9-03cc-41f1-8e8a-0d5df7c72358 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:39:18.274420 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-91a30eb9-03cc-41f1-8e8a-0d5df7c72358 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:39:18.334916 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1476/2939] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:39:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:39:18.346486 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-898e5a11-7b07-415f-8fb7-60319056edfd admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:39:18.347983 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-898e5a11-7b07-415f-8fb7-60319056edfd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:39:18.358212 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:39:18.410263 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-898e5a11-7b07-415f-8fb7-60319056edfd admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:39:18.413993 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0495d8bf-7a10-4099-8ad1-35742c20df4e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:39:18.414612 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0495d8bf-7a10-4099-8ad1-35742c20df4e service neutron] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:39:18.414968 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0495d8bf-7a10-4099-8ad1-35742c20df4e service neutron] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:39:18.415158 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0495d8bf-7a10-4099-8ad1-35742c20df4e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:39:18.424612 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0495d8bf-7a10-4099-8ad1-35742c20df4e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:39:18.426107 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0495d8bf-7a10-4099-8ad1-35742c20df4e service neutron] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:39:18.436616 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1464/2940] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:39:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:39:18.711752 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91a30eb9-03cc-41f1-8e8a-0d5df7c72358 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:39:18.713309 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-91a30eb9-03cc-41f1-8e8a-0d5df7c72358 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:39:18.727096 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91a30eb9-03cc-41f1-8e8a-0d5df7c72358 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:39:18.732056 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf303695-a985-40f5-9a11-9018b5dbd40f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:39:18.732715 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf303695-a985-40f5-9a11-9018b5dbd40f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:39:18.732984 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf303695-a985-40f5-9a11-9018b5dbd40f admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:39:18.733340 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf303695-a985-40f5-9a11-9018b5dbd40f admin admin] PATH_INFO: `/v3/projects/5c3daf077901407095e8648c6032d1b2` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:39:18.734479 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf303695-a985-40f5-9a11-9018b5dbd40f admin admin] RBAC: Authorizing `identity:delete_project(project_id=5c3daf077901407095e8648c6032d1b2)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:39:18.736503 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf303695-a985-40f5-9a11-9018b5dbd40f admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:39:18.761279 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:39:18.788438 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-cf303695-a985-40f5-9a11-9018b5dbd40f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5c3daf077901407095e8648c6032d1b2', 'request_id': 'req-cf303695-a985-40f5-9a11-9018b5dbd40f'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:39:18.794075 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-cf303695-a985-40f5-9a11-9018b5dbd40f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5c3daf077901407095e8648c6032d1b2', 'request_id': 'req-cf303695-a985-40f5-9a11-9018b5dbd40f'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:39:18.800313 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-cf303695-a985-40f5-9a11-9018b5dbd40f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5c3daf077901407095e8648c6032d1b2', 'request_id': 'req-cf303695-a985-40f5-9a11-9018b5dbd40f'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:39:18.817753 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-cf303695-a985-40f5-9a11-9018b5dbd40f admin admin] The token cache is being invalidate because project 5c3daf077901407095e8648c6032d1b2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81508) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 01:39:18.818037 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-cf303695-a985-40f5-9a11-9018b5dbd40f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:39:18.818950 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-cf303695-a985-40f5-9a11-9018b5dbd40f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:39:18.821034 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1477/2941] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 01:39:18 2022] DELETE /identity/v3/projects/5c3daf077901407095e8648c6032d1b2 => generated 0 bytes in 110 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:39:18.825830 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0495d8bf-7a10-4099-8ad1-35742c20df4e service neutron] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:39:18.837205 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0495d8bf-7a10-4099-8ad1-35742c20df4e service neutron] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:39:18.894609 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-798b06d1-c9da-4052-bef9-5192647fe64e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:39:18.895397 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-798b06d1-c9da-4052-bef9-5192647fe64e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:39:18.895660 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-798b06d1-c9da-4052-bef9-5192647fe64e admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:39:18.895932 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-798b06d1-c9da-4052-bef9-5192647fe64e admin admin] PATH_INFO: `/v3/users/682138b0839b4e35b1b46b25d4dfed05` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:39:18.897201 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-798b06d1-c9da-4052-bef9-5192647fe64e admin admin] RBAC: Authorizing `identity:delete_user(user_id=682138b0839b4e35b1b46b25d4dfed05)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:39:18.899187 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-798b06d1-c9da-4052-bef9-5192647fe64e admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:39:18.947609 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-798b06d1-c9da-4052-bef9-5192647fe64e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '682138b0839b4e35b1b46b25d4dfed05'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:39:18.959414 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-798b06d1-c9da-4052-bef9-5192647fe64e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '682138b0839b4e35b1b46b25d4dfed05'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:39:18.967527 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-798b06d1-c9da-4052-bef9-5192647fe64e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '682138b0839b4e35b1b46b25d4dfed05'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:39:18.975146 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-798b06d1-c9da-4052-bef9-5192647fe64e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '682138b0839b4e35b1b46b25d4dfed05'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:39:18.976009 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-798b06d1-c9da-4052-bef9-5192647fe64e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '682138b0839b4e35b1b46b25d4dfed05'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:39:18.982242 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1465/2942] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 01:39:18 2022] DELETE /identity/v3/users/682138b0839b4e35b1b46b25d4dfed05 => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:39:19.268248 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf303695-a985-40f5-9a11-9018b5dbd40f admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:39:19.270071 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-cf303695-a985-40f5-9a11-9018b5dbd40f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:39:19.286980 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf303695-a985-40f5-9a11-9018b5dbd40f admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:39:19.347114 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0779a1b0-7bfb-467e-8ce4-563b741d546c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:39:19.347748 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0779a1b0-7bfb-467e-8ce4-563b741d546c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:39:19.348015 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0779a1b0-7bfb-467e-8ce4-563b741d546c admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:39:19.348290 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0779a1b0-7bfb-467e-8ce4-563b741d546c admin admin] PATH_INFO: `/v3/projects/d836d94222404a828860b25626d5b94f` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:39:19.349472 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0779a1b0-7bfb-467e-8ce4-563b741d546c admin admin] RBAC: Authorizing `identity:delete_project(project_id=d836d94222404a828860b25626d5b94f)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:39:19.351435 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0779a1b0-7bfb-467e-8ce4-563b741d546c admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:39:19.404410 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-0779a1b0-7bfb-467e-8ce4-563b741d546c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd836d94222404a828860b25626d5b94f', 'request_id': 'req-0779a1b0-7bfb-467e-8ce4-563b741d546c'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:39:19.410253 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-0779a1b0-7bfb-467e-8ce4-563b741d546c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd836d94222404a828860b25626d5b94f', 'request_id': 'req-0779a1b0-7bfb-467e-8ce4-563b741d546c'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:39:19.416589 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-0779a1b0-7bfb-467e-8ce4-563b741d546c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd836d94222404a828860b25626d5b94f', 'request_id': 'req-0779a1b0-7bfb-467e-8ce4-563b741d546c'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:39:19.432889 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-0779a1b0-7bfb-467e-8ce4-563b741d546c admin admin] The token cache is being invalidate because project d836d94222404a828860b25626d5b94f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81508) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 01:39:19.433207 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-0779a1b0-7bfb-467e-8ce4-563b741d546c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:39:19.433829 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-0779a1b0-7bfb-467e-8ce4-563b741d546c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:39:19.435334 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1478/2943] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 01:39:19 2022] DELETE /identity/v3/projects/d836d94222404a828860b25626d5b94f => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:39:19.439597 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-798b06d1-c9da-4052-bef9-5192647fe64e admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:39:19.441106 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-798b06d1-c9da-4052-bef9-5192647fe64e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:39:19.454724 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-798b06d1-c9da-4052-bef9-5192647fe64e admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:39:19.482487 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:39:19.514402 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b27d7fec-35fa-4222-8bac-3c534a7f6e37 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:39:19.514959 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b27d7fec-35fa-4222-8bac-3c534a7f6e37 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:39:19.515238 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b27d7fec-35fa-4222-8bac-3c534a7f6e37 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:39:19.515507 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b27d7fec-35fa-4222-8bac-3c534a7f6e37 admin admin] PATH_INFO: `/v3/users/abc37bd535cc47da965107446f03097f` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:39:19.516657 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b27d7fec-35fa-4222-8bac-3c534a7f6e37 admin admin] RBAC: Authorizing `identity:delete_user(user_id=abc37bd535cc47da965107446f03097f)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:39:19.518621 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b27d7fec-35fa-4222-8bac-3c534a7f6e37 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:39:19.568452 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-b27d7fec-35fa-4222-8bac-3c534a7f6e37 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'abc37bd535cc47da965107446f03097f'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:39:19.581247 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-b27d7fec-35fa-4222-8bac-3c534a7f6e37 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'abc37bd535cc47da965107446f03097f'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:39:19.590457 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-b27d7fec-35fa-4222-8bac-3c534a7f6e37 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'abc37bd535cc47da965107446f03097f'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:39:19.597963 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-b27d7fec-35fa-4222-8bac-3c534a7f6e37 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'abc37bd535cc47da965107446f03097f'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:39:19.598643 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-b27d7fec-35fa-4222-8bac-3c534a7f6e37 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'abc37bd535cc47da965107446f03097f'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:39:19.604719 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1466/2944] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 01:39:19 2022] DELETE /identity/v3/users/abc37bd535cc47da965107446f03097f => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:39:19.899760 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0779a1b0-7bfb-467e-8ce4-563b741d546c admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:39:19.901948 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-0779a1b0-7bfb-467e-8ce4-563b741d546c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:39:19.918439 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0779a1b0-7bfb-467e-8ce4-563b741d546c admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:39:19.964245 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:DELETE=14,keystone:SELECT=84,keystone:INSERT=2 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:39:19.983678 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-874302eb-5f57-4fd9-a077-389f139d0d9e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:39:19.984301 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-874302eb-5f57-4fd9-a077-389f139d0d9e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:39:19.984609 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-874302eb-5f57-4fd9-a077-389f139d0d9e admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:39:19.985001 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-874302eb-5f57-4fd9-a077-389f139d0d9e admin admin] PATH_INFO: `/v3/projects/d9b1ad09c7ba48e0b004eceab7d81069` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:39:19.986023 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-874302eb-5f57-4fd9-a077-389f139d0d9e admin admin] RBAC: Authorizing `identity:delete_project(project_id=d9b1ad09c7ba48e0b004eceab7d81069)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:39:19.987930 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-874302eb-5f57-4fd9-a077-389f139d0d9e admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:39:20.037318 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-874302eb-5f57-4fd9-a077-389f139d0d9e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd9b1ad09c7ba48e0b004eceab7d81069', 'request_id': 'req-874302eb-5f57-4fd9-a077-389f139d0d9e'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:39:20.043251 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-874302eb-5f57-4fd9-a077-389f139d0d9e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd9b1ad09c7ba48e0b004eceab7d81069', 'request_id': 'req-874302eb-5f57-4fd9-a077-389f139d0d9e'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:39:20.054448 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-874302eb-5f57-4fd9-a077-389f139d0d9e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd9b1ad09c7ba48e0b004eceab7d81069', 'request_id': 'req-874302eb-5f57-4fd9-a077-389f139d0d9e'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:39:20.070045 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-874302eb-5f57-4fd9-a077-389f139d0d9e admin admin] The token cache is being invalidate because project d9b1ad09c7ba48e0b004eceab7d81069 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81508) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 01:39:20.070326 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-874302eb-5f57-4fd9-a077-389f139d0d9e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:39:20.070970 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-874302eb-5f57-4fd9-a077-389f139d0d9e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:39:20.072562 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1479/2945] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 01:39:19 2022] DELETE /identity/v3/projects/d9b1ad09c7ba48e0b004eceab7d81069 => generated 0 bytes in 174 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:39:29.601407 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=40,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:39:30.067477 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=33,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:39:34.379858 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b27d7fec-35fa-4222-8bac-3c534a7f6e37 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:39:34.381463 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-b27d7fec-35fa-4222-8bac-3c534a7f6e37 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:39:34.475042 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b27d7fec-35fa-4222-8bac-3c534a7f6e37 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:39:34.544490 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3ec0efc-108d-421b-a701-f43de082302b service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2a3834fa6894dada3291e4781bbd553', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:39:34.545120 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3ec0efc-108d-421b-a701-f43de082302b service placement] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:39:34.545422 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3ec0efc-108d-421b-a701-f43de082302b service placement] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:39:34.545692 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3ec0efc-108d-421b-a701-f43de082302b service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:39:34.547778 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-e3ec0efc-108d-421b-a701-f43de082302b service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:39:34.560995 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3ec0efc-108d-421b-a701-f43de082302b service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:39:34.562624 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3ec0efc-108d-421b-a701-f43de082302b service placement] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:39:34.574795 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1467/2946] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:39:34 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 196 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:39:44.569794 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=63 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:41:04.632292 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-874302eb-5f57-4fd9-a077-389f139d0d9e admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:41:04.634080 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-874302eb-5f57-4fd9-a077-389f139d0d9e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:41:04.714821 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-874302eb-5f57-4fd9-a077-389f139d0d9e admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:41:04.721169 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19207d1a-b6dc-4430-af59-8368e6d15098 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:41:04.721833 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19207d1a-b6dc-4430-af59-8368e6d15098 service neutron] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:41:04.722272 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19207d1a-b6dc-4430-af59-8368e6d15098 service neutron] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:41:04.722807 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19207d1a-b6dc-4430-af59-8368e6d15098 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:41:04.725848 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-19207d1a-b6dc-4430-af59-8368e6d15098 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:41:04.800832 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19207d1a-b6dc-4430-af59-8368e6d15098 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:41:04.802323 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19207d1a-b6dc-4430-af59-8368e6d15098 service neutron] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:41:04.813966 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1480/2947] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:41:04 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 182 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:41:05.271249 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3ec0efc-108d-421b-a701-f43de082302b service placement] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:41:05.275998 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-e3ec0efc-108d-421b-a701-f43de082302b service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:41:05.305997 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3ec0efc-108d-421b-a701-f43de082302b service placement] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:41:05.315001 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b716c51-6982-4189-9a18-0f6baa82f353 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '168b07df50c24593ba73dbde5b037aed', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:41:05.315753 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b716c51-6982-4189-9a18-0f6baa82f353 service nova] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:41:05.316154 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b716c51-6982-4189-9a18-0f6baa82f353 service nova] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:41:05.316574 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b716c51-6982-4189-9a18-0f6baa82f353 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:41:05.319502 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-3b716c51-6982-4189-9a18-0f6baa82f353 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:41:05.341238 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b716c51-6982-4189-9a18-0f6baa82f353 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:41:05.343464 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b716c51-6982-4189-9a18-0f6baa82f353 service nova] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:41:05.357586 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1468/2948] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:41:05 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:41:09.089029 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19207d1a-b6dc-4430-af59-8368e6d15098 service neutron] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:41:09.091166 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-19207d1a-b6dc-4430-af59-8368e6d15098 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:41:09.168839 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19207d1a-b6dc-4430-af59-8368e6d15098 service neutron] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:41:09.173210 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cd65b05-e9b6-4234-8d36-e3717b5ad110 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '879afc99ebb9402b94fadc6091040c5c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:41:09.173824 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cd65b05-e9b6-4234-8d36-e3717b5ad110 service cinder] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:41:09.174102 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cd65b05-e9b6-4234-8d36-e3717b5ad110 service cinder] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:41:09.174371 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cd65b05-e9b6-4234-8d36-e3717b5ad110 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:41:09.176582 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-0cd65b05-e9b6-4234-8d36-e3717b5ad110 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:41:09.194178 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cd65b05-e9b6-4234-8d36-e3717b5ad110 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:41:09.196207 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cd65b05-e9b6-4234-8d36-e3717b5ad110 service cinder] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:41:09.207556 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1481/2949] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:41:09 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 119 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:41:14.509511 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce169fe7-2140-4f58-a183-50b3004a91d2 None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:41:14.510007 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce169fe7-2140-4f58-a183-50b3004a91d2 None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:41:14.510100 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce169fe7-2140-4f58-a183-50b3004a91d2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:41:14.538970 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-ce169fe7-2140-4f58-a183-50b3004a91d2 None None] MFA Rules not processed for user `eede08f020e04282bb994b855e371433`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:41:14.540381 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-ce169fe7-2140-4f58-a183-50b3004a91d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:41:14.615075 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1469/2950] 10.222.0.42 () {58 vars in 967 bytes} [Sun Aug 28 01:41:14 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:41:14.663527 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cd65b05-e9b6-4234-8d36-e3717b5ad110 service cinder] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:41:14.665218 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-0cd65b05-e9b6-4234-8d36-e3717b5ad110 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:41:14.701667 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=100 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:41:14.739993 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cd65b05-e9b6-4234-8d36-e3717b5ad110 service cinder] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:41:14.746600 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fda2836d-4ba5-4c21-8df0-419e508174db service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ac4c5584582f4f55a7085af4cbbabf9b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:41:14.747210 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda2836d-4ba5-4c21-8df0-419e508174db service swift] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:41:14.747490 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda2836d-4ba5-4c21-8df0-419e508174db service swift] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:41:14.747758 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda2836d-4ba5-4c21-8df0-419e508174db service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:41:14.757092 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fda2836d-4ba5-4c21-8df0-419e508174db service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:41:14.758892 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fda2836d-4ba5-4c21-8df0-419e508174db service swift] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:41:14.766950 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1482/2951] 10.222.0.42 () {60 vars in 1384 bytes} [Sun Aug 28 01:41:14 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Aug 28 01:41:21.264034 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce169fe7-2140-4f58-a183-50b3004a91d2 None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:41:21.265825 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-ce169fe7-2140-4f58-a183-50b3004a91d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:41:21.283528 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce169fe7-2140-4f58-a183-50b3004a91d2 None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:41:21.289736 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9007951-c629-4149-a6a8-c1384c40ba5b service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2a3834fa6894dada3291e4781bbd553', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:41:21.290422 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9007951-c629-4149-a6a8-c1384c40ba5b service placement] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:41:21.290703 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9007951-c629-4149-a6a8-c1384c40ba5b service placement] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:41:21.290978 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9007951-c629-4149-a6a8-c1384c40ba5b service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:41:21.292793 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-a9007951-c629-4149-a6a8-c1384c40ba5b service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:41:21.306713 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9007951-c629-4149-a6a8-c1384c40ba5b service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:41:21.308805 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9007951-c629-4149-a6a8-c1384c40ba5b service placement] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:41:21.321074 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1470/2952] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:41:21 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 58 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:41:21.322825 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fda2836d-4ba5-4c21-8df0-419e508174db service swift] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:41:21.325152 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-fda2836d-4ba5-4c21-8df0-419e508174db service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:41:21.432231 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fda2836d-4ba5-4c21-8df0-419e508174db service swift] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:41:21.438735 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-090c3702-c327-4a7c-8059-4fc9b35c0869 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:41:21.439547 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-090c3702-c327-4a7c-8059-4fc9b35c0869 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:41:21.439693 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-090c3702-c327-4a7c-8059-4fc9b35c0869 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:41:21.439897 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-090c3702-c327-4a7c-8059-4fc9b35c0869 admin admin] PATH_INFO: `/v3/users/c5bc73cffe1e4fa095051e109b5f0d0c` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:41:21.441205 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-090c3702-c327-4a7c-8059-4fc9b35c0869 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c5bc73cffe1e4fa095051e109b5f0d0c)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:41:21.443203 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-090c3702-c327-4a7c-8059-4fc9b35c0869 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:41:21.496008 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-090c3702-c327-4a7c-8059-4fc9b35c0869 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c5bc73cffe1e4fa095051e109b5f0d0c'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:41:21.496593 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-090c3702-c327-4a7c-8059-4fc9b35c0869 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c5bc73cffe1e4fa095051e109b5f0d0c'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:41:21.505536 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-090c3702-c327-4a7c-8059-4fc9b35c0869 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c5bc73cffe1e4fa095051e109b5f0d0c'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:41:21.512818 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-090c3702-c327-4a7c-8059-4fc9b35c0869 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c5bc73cffe1e4fa095051e109b5f0d0c'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:41:21.513490 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-090c3702-c327-4a7c-8059-4fc9b35c0869 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c5bc73cffe1e4fa095051e109b5f0d0c'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:41:21.530686 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1483/2953] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 01:41:21 2022] DELETE /identity/v3/users/c5bc73cffe1e4fa095051e109b5f0d0c => generated 0 bytes in 209 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:41:21.703605 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9007951-c629-4149-a6a8-c1384c40ba5b service placement] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:41:21.714444 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9007951-c629-4149-a6a8-c1384c40ba5b service placement] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:41:21.776360 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d84ff19-da32-435c-82ca-156329a8136b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:41:21.777055 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d84ff19-da32-435c-82ca-156329a8136b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:41:21.777335 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d84ff19-da32-435c-82ca-156329a8136b admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:41:21.777603 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d84ff19-da32-435c-82ca-156329a8136b admin admin] PATH_INFO: `/v3/projects/8bd4256c51f64ef9aa0f21ea2fcb55e7` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:41:21.779036 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d84ff19-da32-435c-82ca-156329a8136b admin admin] RBAC: Authorizing `identity:delete_project(project_id=8bd4256c51f64ef9aa0f21ea2fcb55e7)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:41:21.781032 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d84ff19-da32-435c-82ca-156329a8136b admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:41:21.840830 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-8d84ff19-da32-435c-82ca-156329a8136b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8bd4256c51f64ef9aa0f21ea2fcb55e7', 'request_id': 'req-8d84ff19-da32-435c-82ca-156329a8136b'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:41:21.847379 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-8d84ff19-da32-435c-82ca-156329a8136b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8bd4256c51f64ef9aa0f21ea2fcb55e7', 'request_id': 'req-8d84ff19-da32-435c-82ca-156329a8136b'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:41:21.854006 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-8d84ff19-da32-435c-82ca-156329a8136b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8bd4256c51f64ef9aa0f21ea2fcb55e7', 'request_id': 'req-8d84ff19-da32-435c-82ca-156329a8136b'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:41:21.868849 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-8d84ff19-da32-435c-82ca-156329a8136b admin admin] The token cache is being invalidate because project 8bd4256c51f64ef9aa0f21ea2fcb55e7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81507) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 01:41:21.869164 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-8d84ff19-da32-435c-82ca-156329a8136b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:41:21.869880 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-8d84ff19-da32-435c-82ca-156329a8136b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:41:21.871476 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1471/2954] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 01:41:21 2022] DELETE /identity/v3/projects/8bd4256c51f64ef9aa0f21ea2fcb55e7 => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:41:21.875952 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-090c3702-c327-4a7c-8059-4fc9b35c0869 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:41:21.877940 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-090c3702-c327-4a7c-8059-4fc9b35c0869 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:41:21.897601 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-090c3702-c327-4a7c-8059-4fc9b35c0869 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:41:21.907306 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:41:21.959993 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94426398-f03c-4b31-8931-94b65f7e51bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:41:21.960742 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94426398-f03c-4b31-8931-94b65f7e51bb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:41:21.961019 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94426398-f03c-4b31-8931-94b65f7e51bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:41:21.961320 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94426398-f03c-4b31-8931-94b65f7e51bb admin admin] PATH_INFO: `/v3/users/300220ede3bc4d67a443eeab0d6511b2` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:41:21.962538 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94426398-f03c-4b31-8931-94b65f7e51bb admin admin] RBAC: Authorizing `identity:delete_user(user_id=300220ede3bc4d67a443eeab0d6511b2)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:41:21.964482 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94426398-f03c-4b31-8931-94b65f7e51bb admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:41:22.022246 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-94426398-f03c-4b31-8931-94b65f7e51bb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '300220ede3bc4d67a443eeab0d6511b2'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:41:22.022929 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-94426398-f03c-4b31-8931-94b65f7e51bb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '300220ede3bc4d67a443eeab0d6511b2'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:41:22.032890 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-94426398-f03c-4b31-8931-94b65f7e51bb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '300220ede3bc4d67a443eeab0d6511b2'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:41:22.040510 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-94426398-f03c-4b31-8931-94b65f7e51bb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '300220ede3bc4d67a443eeab0d6511b2'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:41:22.041179 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-94426398-f03c-4b31-8931-94b65f7e51bb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '300220ede3bc4d67a443eeab0d6511b2'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:41:22.058991 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1484/2955] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 01:41:21 2022] DELETE /identity/v3/users/300220ede3bc4d67a443eeab0d6511b2 => generated 0 bytes in 184 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:41:22.244474 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d84ff19-da32-435c-82ca-156329a8136b admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:41:22.247350 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-8d84ff19-da32-435c-82ca-156329a8136b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:41:22.263789 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d84ff19-da32-435c-82ca-156329a8136b admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:41:22.269843 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:41:22.328938 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12ec2231-46a2-4fed-8dd4-2156d896d1af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:41:22.329608 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ec2231-46a2-4fed-8dd4-2156d896d1af admin admin] REQUEST_METHOD: `DELETE` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:41:22.329876 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ec2231-46a2-4fed-8dd4-2156d896d1af admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:41:22.330158 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ec2231-46a2-4fed-8dd4-2156d896d1af admin admin] PATH_INFO: `/v3/projects/52b4d06d235646a8a3b7074b559dba9c` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:41:22.331297 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12ec2231-46a2-4fed-8dd4-2156d896d1af admin admin] RBAC: Authorizing `identity:delete_project(project_id=52b4d06d235646a8a3b7074b559dba9c)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:41:22.333251 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12ec2231-46a2-4fed-8dd4-2156d896d1af admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:41:22.383354 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-12ec2231-46a2-4fed-8dd4-2156d896d1af admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '52b4d06d235646a8a3b7074b559dba9c', 'request_id': 'req-12ec2231-46a2-4fed-8dd4-2156d896d1af'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:41:22.388739 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-12ec2231-46a2-4fed-8dd4-2156d896d1af admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '52b4d06d235646a8a3b7074b559dba9c', 'request_id': 'req-12ec2231-46a2-4fed-8dd4-2156d896d1af'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:41:22.394895 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-12ec2231-46a2-4fed-8dd4-2156d896d1af admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '52b4d06d235646a8a3b7074b559dba9c', 'request_id': 'req-12ec2231-46a2-4fed-8dd4-2156d896d1af'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:41:22.411343 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-12ec2231-46a2-4fed-8dd4-2156d896d1af admin admin] The token cache is being invalidate because project 52b4d06d235646a8a3b7074b559dba9c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81507) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 01:41:22.411630 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-12ec2231-46a2-4fed-8dd4-2156d896d1af admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:41:22.412256 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-12ec2231-46a2-4fed-8dd4-2156d896d1af admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:41:22.413782 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1472/2956] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 01:41:22 2022] DELETE /identity/v3/projects/52b4d06d235646a8a3b7074b559dba9c => generated 0 bytes in 171 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 28 01:41:22.419847 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0de7ba6c-676d-4b92-ae94-a51fd3495139 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:41:22.420352 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0de7ba6c-676d-4b92-ae94-a51fd3495139 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:41:22.420529 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0de7ba6c-676d-4b92-ae94-a51fd3495139 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:41:22.485859 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-0de7ba6c-676d-4b92-ae94-a51fd3495139 None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:41:22.487080 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-0de7ba6c-676d-4b92-ae94-a51fd3495139 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:41:22.554149 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1485/2957] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:41:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:41:22.562818 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b588372-6efb-4c07-a8f2-5682ffbbbfbe None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:41:22.563100 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b588372-6efb-4c07-a8f2-5682ffbbbfbe None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:41:22.563372 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b588372-6efb-4c07-a8f2-5682ffbbbfbe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:41:22.583453 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-0b588372-6efb-4c07-a8f2-5682ffbbbfbe None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:41:22.584665 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-0b588372-6efb-4c07-a8f2-5682ffbbbfbe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:41:22.593208 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1473/2958] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:41:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:41:22.597707 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0de7ba6c-676d-4b92-ae94-a51fd3495139 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:41:22.602668 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:41:22.609263 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0de7ba6c-676d-4b92-ae94-a51fd3495139 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:41:22.615691 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da1146e5-f09a-4bb2-b6f7-fb1e45498eba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:41:22.616285 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da1146e5-f09a-4bb2-b6f7-fb1e45498eba admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:41:22.616553 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da1146e5-f09a-4bb2-b6f7-fb1e45498eba admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:41:22.616815 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da1146e5-f09a-4bb2-b6f7-fb1e45498eba admin admin] PATH_INFO: `/v3/domains` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:41:22.617480 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da1146e5-f09a-4bb2-b6f7-fb1e45498eba admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81508) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Aug 28 01:41:22.617770 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da1146e5-f09a-4bb2-b6f7-fb1e45498eba admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:41:22.619423 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da1146e5-f09a-4bb2-b6f7-fb1e45498eba admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:41:22.631556 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1486/2959] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Aug 28 01:41:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Aug 28 01:41:22.636208 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b588372-6efb-4c07-a8f2-5682ffbbbfbe None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:41:22.645950 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b588372-6efb-4c07-a8f2-5682ffbbbfbe None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:41:22.651842 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eb20970-9207-4bce-bbce-14adceb97cce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:41:22.652441 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eb20970-9207-4bce-bbce-14adceb97cce admin admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:41:22.652715 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eb20970-9207-4bce-bbce-14adceb97cce admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:41:22.652995 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eb20970-9207-4bce-bbce-14adceb97cce admin admin] PATH_INFO: `/v3/projects` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:41:22.653646 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eb20970-9207-4bce-bbce-14adceb97cce admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:41:22.655551 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eb20970-9207-4bce-bbce-14adceb97cce admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:41:22.700113 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1474/2960] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 01:41:22 2022] POST /identity/v3/projects => generated 373 bytes in 64 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 28 01:41:22.704712 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da1146e5-f09a-4bb2-b6f7-fb1e45498eba admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:41:22.715255 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da1146e5-f09a-4bb2-b6f7-fb1e45498eba admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:41:22.785186 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1405e168-c283-42a7-b9db-6ce58d192adf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:41:22.786147 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1405e168-c283-42a7-b9db-6ce58d192adf admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:41:22.786639 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1405e168-c283-42a7-b9db-6ce58d192adf admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:41:22.787072 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1405e168-c283-42a7-b9db-6ce58d192adf admin admin] PATH_INFO: `/v3/users` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:41:22.788059 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1405e168-c283-42a7-b9db-6ce58d192adf admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:41:22.790037 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1405e168-c283-42a7-b9db-6ce58d192adf admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:41:22.818589 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1487/2961] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 01:41:22 2022] POST /identity/v3/users => generated 287 bytes in 115 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 01:41:22.823053 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eb20970-9207-4bce-bbce-14adceb97cce admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:41:22.834075 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eb20970-9207-4bce-bbce-14adceb97cce admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:41:22.839912 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-518883ca-8a7b-4213-8aed-56282421993b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:41:22.840517 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-518883ca-8a7b-4213-8aed-56282421993b admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:41:22.840792 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-518883ca-8a7b-4213-8aed-56282421993b admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:41:22.841096 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-518883ca-8a7b-4213-8aed-56282421993b admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:41:22.841744 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-518883ca-8a7b-4213-8aed-56282421993b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:41:22.843362 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-518883ca-8a7b-4213-8aed-56282421993b admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:41:22.854641 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1475/2962] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 01:41:22 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 01:41:22.859625 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1405e168-c283-42a7-b9db-6ce58d192adf admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:41:22.876479 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1405e168-c283-42a7-b9db-6ce58d192adf admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:41:22.884648 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6798d4b8-5dbb-48aa-bf1a-6018e93c78e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:41:22.885276 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6798d4b8-5dbb-48aa-bf1a-6018e93c78e9 admin admin] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:41:22.885566 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6798d4b8-5dbb-48aa-bf1a-6018e93c78e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:41:22.885800 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6798d4b8-5dbb-48aa-bf1a-6018e93c78e9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:41:22.886544 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6798d4b8-5dbb-48aa-bf1a-6018e93c78e9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:41:22.888144 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6798d4b8-5dbb-48aa-bf1a-6018e93c78e9 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:41:22.904112 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1488/2963] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 01:41:22 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 01:41:22.908533 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-518883ca-8a7b-4213-8aed-56282421993b admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:41:22.919525 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-518883ca-8a7b-4213-8aed-56282421993b admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:41:22.925690 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cea43b7-6ecf-46e1-9337-ce6aea0c6507 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:41:22.926374 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cea43b7-6ecf-46e1-9337-ce6aea0c6507 admin admin] REQUEST_METHOD: `PUT` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:41:22.926652 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cea43b7-6ecf-46e1-9337-ce6aea0c6507 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:41:22.926919 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cea43b7-6ecf-46e1-9337-ce6aea0c6507 admin admin] PATH_INFO: `/v3/projects/b4ead68269cb4e19befec4146c93686d/users/41a51cf16450452194671aa46f4f06c4/roles/0487ef7e943e4701915cb8ce0f2899d7` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:41:22.942850 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cea43b7-6ecf-46e1-9337-ce6aea0c6507 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b4ead68269cb4e19befec4146c93686d, user_id=41a51cf16450452194671aa46f4f06c4, role_id=0487ef7e943e4701915cb8ce0f2899d7)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:41:22.945600 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cea43b7-6ecf-46e1-9337-ce6aea0c6507 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:41:22.961274 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1476/2964] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 01:41:22 2022] PUT /identity/v3/projects/b4ead68269cb4e19befec4146c93686d/users/41a51cf16450452194671aa46f4f06c4/roles/0487ef7e943e4701915cb8ce0f2899d7 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:41:22.966369 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a9cf00d-23c8-489c-a42c-aaf3855d12df None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:41:22.966662 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a9cf00d-23c8-489c-a42c-aaf3855d12df None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:41:22.966955 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a9cf00d-23c8-489c-a42c-aaf3855d12df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:41:22.987116 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-4a9cf00d-23c8-489c-a42c-aaf3855d12df None None] MFA Rules not processed for user `e5cb815a43fc4460b6ddf0175f087128`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:41:22.988349 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-4a9cf00d-23c8-489c-a42c-aaf3855d12df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:41:23.052510 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1489/2965] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:41:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:41:23.063101 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cea43b7-6ecf-46e1-9337-ce6aea0c6507 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:41:23.065089 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-5cea43b7-6ecf-46e1-9337-ce6aea0c6507 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:41:23.107188 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:41:23.152920 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cea43b7-6ecf-46e1-9337-ce6aea0c6507 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:41:23.156816 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f490d8ed-2de8-4984-843a-55ed2f9f092c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:41:23.157438 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f490d8ed-2de8-4984-843a-55ed2f9f092c service neutron] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:41:23.157707 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f490d8ed-2de8-4984-843a-55ed2f9f092c service neutron] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:41:23.157977 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f490d8ed-2de8-4984-843a-55ed2f9f092c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:41:23.167703 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f490d8ed-2de8-4984-843a-55ed2f9f092c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:41:23.169167 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f490d8ed-2de8-4984-843a-55ed2f9f092c service neutron] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:41:23.178990 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1477/2966] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:41:23 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 116 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:41:27.209562 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5411199-1639-4ad3-8058-6b6f2f1483a0 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:41:27.210345 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5411199-1639-4ad3-8058-6b6f2f1483a0 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:41:27.210499 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5411199-1639-4ad3-8058-6b6f2f1483a0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:41:27.246117 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-b5411199-1639-4ad3-8058-6b6f2f1483a0 None None] MFA Rules not processed for user `41a51cf16450452194671aa46f4f06c4`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:41:27.247368 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-b5411199-1639-4ad3-8058-6b6f2f1483a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:41:27.330247 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:41:27.336848 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1490/2967] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:41:27 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:41:27.342588 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f490d8ed-2de8-4984-843a-55ed2f9f092c service neutron] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:41:27.352864 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f490d8ed-2de8-4984-843a-55ed2f9f092c service neutron] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:41:27.357727 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daf56167-2151-4c42-8798-f06c64a54ad4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:41:27.358339 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daf56167-2151-4c42-8798-f06c64a54ad4 admin admin] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:41:27.358650 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daf56167-2151-4c42-8798-f06c64a54ad4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:41:27.358915 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daf56167-2151-4c42-8798-f06c64a54ad4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:41:27.359545 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daf56167-2151-4c42-8798-f06c64a54ad4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:41:27.361756 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daf56167-2151-4c42-8798-f06c64a54ad4 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:41:27.392391 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1478/2968] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Aug 28 01:41:27 2022] POST /identity/v3/projects => generated 373 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Aug 28 01:41:27.396719 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5411199-1639-4ad3-8058-6b6f2f1483a0 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:41:27.408636 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5411199-1639-4ad3-8058-6b6f2f1483a0 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:41:27.467269 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5427b73d-1b02-402b-a957-124e1c9184fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:41:27.467894 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5427b73d-1b02-402b-a957-124e1c9184fc admin admin] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:41:27.468170 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5427b73d-1b02-402b-a957-124e1c9184fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:41:27.468553 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5427b73d-1b02-402b-a957-124e1c9184fc admin admin] PATH_INFO: `/v3/users` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:41:27.469134 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5427b73d-1b02-402b-a957-124e1c9184fc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:41:27.471064 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5427b73d-1b02-402b-a957-124e1c9184fc admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:41:27.496646 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1491/2969] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Aug 28 01:41:27 2022] POST /identity/v3/users => generated 293 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Aug 28 01:41:27.500831 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daf56167-2151-4c42-8798-f06c64a54ad4 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:41:27.511897 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daf56167-2151-4c42-8798-f06c64a54ad4 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:41:27.516712 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10b64435-2016-4204-9001-c48a20827b71 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:41:27.517295 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10b64435-2016-4204-9001-c48a20827b71 admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:41:27.517571 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10b64435-2016-4204-9001-c48a20827b71 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:41:27.517836 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10b64435-2016-4204-9001-c48a20827b71 admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:41:27.518476 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10b64435-2016-4204-9001-c48a20827b71 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:41:27.520120 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10b64435-2016-4204-9001-c48a20827b71 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:41:27.532040 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1479/2970] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 01:41:27 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 01:41:27.536085 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5427b73d-1b02-402b-a957-124e1c9184fc admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:41:27.546740 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5427b73d-1b02-402b-a957-124e1c9184fc admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:41:27.551771 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6fdb070-3477-4b58-b760-7c5a03331f51 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:41:27.552452 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6fdb070-3477-4b58-b760-7c5a03331f51 admin admin] REQUEST_METHOD: `PUT` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:41:27.552729 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6fdb070-3477-4b58-b760-7c5a03331f51 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:41:27.553054 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6fdb070-3477-4b58-b760-7c5a03331f51 admin admin] PATH_INFO: `/v3/projects/c1e59266819e4570885584a9bd3e031d/users/e2091185562548218ab23c4797dcded8/roles/f08d6043261747e3b4763c744eef7dfb` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:41:27.570696 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6fdb070-3477-4b58-b760-7c5a03331f51 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c1e59266819e4570885584a9bd3e031d, user_id=e2091185562548218ab23c4797dcded8, role_id=f08d6043261747e3b4763c744eef7dfb)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:41:27.573562 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6fdb070-3477-4b58-b760-7c5a03331f51 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:41:27.589936 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1492/2971] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 01:41:27 2022] PUT /identity/v3/projects/c1e59266819e4570885584a9bd3e031d/users/e2091185562548218ab23c4797dcded8/roles/f08d6043261747e3b4763c744eef7dfb => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:41:27.593926 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10b64435-2016-4204-9001-c48a20827b71 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:41:27.604798 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10b64435-2016-4204-9001-c48a20827b71 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:41:27.666991 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a4a610a-8a10-49e0-8858-441ffb93f889 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:41:27.667592 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a4a610a-8a10-49e0-8858-441ffb93f889 admin admin] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:41:27.667869 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a4a610a-8a10-49e0-8858-441ffb93f889 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:41:27.668142 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a4a610a-8a10-49e0-8858-441ffb93f889 admin admin] PATH_INFO: `/v3/roles` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:41:27.668780 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a4a610a-8a10-49e0-8858-441ffb93f889 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:41:27.670486 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a4a610a-8a10-49e0-8858-441ffb93f889 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:41:27.691273 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1480/2972] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Aug 28 01:41:27 2022] GET /identity/v3/roles => generated 1420 bytes in 92 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Aug 28 01:41:27.691516 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6fdb070-3477-4b58-b760-7c5a03331f51 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:41:27.708180 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6fdb070-3477-4b58-b760-7c5a03331f51 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:41:27.714767 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34a8be85-015a-444d-a1da-83f6db810c44 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:41:27.715437 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34a8be85-015a-444d-a1da-83f6db810c44 admin admin] REQUEST_METHOD: `PUT` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:41:27.715714 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34a8be85-015a-444d-a1da-83f6db810c44 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:41:27.715984 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34a8be85-015a-444d-a1da-83f6db810c44 admin admin] PATH_INFO: `/v3/projects/c1e59266819e4570885584a9bd3e031d/users/e2091185562548218ab23c4797dcded8/roles/f08d6043261747e3b4763c744eef7dfb` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:41:27.717967 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34a8be85-015a-444d-a1da-83f6db810c44 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c1e59266819e4570885584a9bd3e031d, user_id=e2091185562548218ab23c4797dcded8, role_id=f08d6043261747e3b4763c744eef7dfb)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:41:27.720636 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34a8be85-015a-444d-a1da-83f6db810c44 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:41:27.736868 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1493/2973] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Aug 28 01:41:27 2022] PUT /identity/v3/projects/c1e59266819e4570885584a9bd3e031d/users/e2091185562548218ab23c4797dcded8/roles/f08d6043261747e3b4763c744eef7dfb => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Aug 28 01:41:32.329222 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f9d643-7604-4598-b084-b361179f176a None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:41:32.329818 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f9d643-7604-4598-b084-b361179f176a None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:41:32.329818 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f9d643-7604-4598-b084-b361179f176a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:41:32.363680 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-05f9d643-7604-4598-b084-b361179f176a None None] MFA Rules not processed for user `e2091185562548218ab23c4797dcded8`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:41:32.364928 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-05f9d643-7604-4598-b084-b361179f176a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:41:32.445220 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1481/2974] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:41:32 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:41:32.460385 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34a8be85-015a-444d-a1da-83f6db810c44 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:41:32.464158 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-34a8be85-015a-444d-a1da-83f6db810c44 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:41:32.556926 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34a8be85-015a-444d-a1da-83f6db810c44 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:41:32.563356 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fec79418-3797-4fdb-b040-b7508bd0cfb6 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '168b07df50c24593ba73dbde5b037aed', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:41:32.563988 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec79418-3797-4fdb-b040-b7508bd0cfb6 service nova] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:41:32.564265 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec79418-3797-4fdb-b040-b7508bd0cfb6 service nova] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:41:32.564535 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec79418-3797-4fdb-b040-b7508bd0cfb6 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:41:32.616654 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:41:32.644925 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fec79418-3797-4fdb-b040-b7508bd0cfb6 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:41:32.646709 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fec79418-3797-4fdb-b040-b7508bd0cfb6 service nova] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:41:32.660102 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1494/2975] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:41:32 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 200 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:41:33.817580 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a97cf62e-0fb6-479d-a458-63ab9a06621e None None] REQUEST_METHOD: `POST` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:41:33.818233 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a97cf62e-0fb6-479d-a458-63ab9a06621e None None] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:41:33.818884 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a97cf62e-0fb6-479d-a458-63ab9a06621e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:41:33.838887 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:41:33.851177 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.auth.core [None req-a97cf62e-0fb6-479d-a458-63ab9a06621e None None] MFA Rules not processed for user `41a51cf16450452194671aa46f4f06c4`. Rule list: `[]` (Enabled: `True`). {{(pid=81507) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:41:33.852909 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-a97cf62e-0fb6-479d-a458-63ab9a06621e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:41:33.863797 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1482/2976] 10.222.0.42 () {58 vars in 958 bytes} [Sun Aug 28 01:41:33 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 48 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:41:42.652412 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=6 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:41:58.036991 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fec79418-3797-4fdb-b040-b7508bd0cfb6 service nova] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:41:58.039472 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-fec79418-3797-4fdb-b040-b7508bd0cfb6 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:41:58.107417 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fec79418-3797-4fdb-b040-b7508bd0cfb6 service nova] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:41:58.112769 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e07a3c36-6623-4b38-a437-9afd61172add service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:41:58.113411 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e07a3c36-6623-4b38-a437-9afd61172add service neutron] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:41:58.113685 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e07a3c36-6623-4b38-a437-9afd61172add service neutron] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:41:58.114530 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e07a3c36-6623-4b38-a437-9afd61172add service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:41:58.115991 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-e07a3c36-6623-4b38-a437-9afd61172add service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:41:58.127259 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e07a3c36-6623-4b38-a437-9afd61172add service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:41:58.128792 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e07a3c36-6623-4b38-a437-9afd61172add service neutron] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:41:58.137911 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1495/2977] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:41:58 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:41:58.656088 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a97cf62e-0fb6-479d-a458-63ab9a06621e None None] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:41:58.657738 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-a97cf62e-0fb6-479d-a458-63ab9a06621e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:41:58.748920 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a97cf62e-0fb6-479d-a458-63ab9a06621e None None] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:41:58.754327 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3335dcc-ff6d-424e-b7a7-169d041f99a8 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2a3834fa6894dada3291e4781bbd553', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:41:58.754873 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3335dcc-ff6d-424e-b7a7-169d041f99a8 service placement] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:41:58.755169 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3335dcc-ff6d-424e-b7a7-169d041f99a8 service placement] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:41:58.755465 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3335dcc-ff6d-424e-b7a7-169d041f99a8 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:41:58.757292 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-e3335dcc-ff6d-424e-b7a7-169d041f99a8 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:41:58.770853 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3335dcc-ff6d-424e-b7a7-169d041f99a8 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:41:58.772830 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3335dcc-ff6d-424e-b7a7-169d041f99a8 service placement] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:41:58.785170 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1483/2978] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:41:58 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 130 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:42:00.756293 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e07a3c36-6623-4b38-a437-9afd61172add service neutron] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:42:00.771784 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e07a3c36-6623-4b38-a437-9afd61172add service neutron] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:42:00.777236 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61469e99-09e5-4899-9a6a-8e06885eac43 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:42:00.777966 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61469e99-09e5-4899-9a6a-8e06885eac43 service neutron] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:42:00.778321 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61469e99-09e5-4899-9a6a-8e06885eac43 service neutron] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:42:00.778688 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61469e99-09e5-4899-9a6a-8e06885eac43 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:42:00.780612 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-61469e99-09e5-4899-9a6a-8e06885eac43 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:42:00.792306 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61469e99-09e5-4899-9a6a-8e06885eac43 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:42:00.793845 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61469e99-09e5-4899-9a6a-8e06885eac43 service neutron] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:42:00.806705 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1496/2979] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:42:00 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 51 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:42:08.780294 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=32 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:42:10.799263 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=32 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:42:13.924801 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3335dcc-ff6d-424e-b7a7-169d041f99a8 service placement] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:42:13.938322 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3335dcc-ff6d-424e-b7a7-169d041f99a8 service placement] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:42:13.944438 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26d877a1-74a9-4bf5-aa2e-7a09c3572e2b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '168b07df50c24593ba73dbde5b037aed', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:42:13.945103 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26d877a1-74a9-4bf5-aa2e-7a09c3572e2b service nova] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:42:13.945380 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26d877a1-74a9-4bf5-aa2e-7a09c3572e2b service nova] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:42:13.945648 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26d877a1-74a9-4bf5-aa2e-7a09c3572e2b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:42:13.947864 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-26d877a1-74a9-4bf5-aa2e-7a09c3572e2b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:42:13.962010 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26d877a1-74a9-4bf5-aa2e-7a09c3572e2b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:42:13.963615 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26d877a1-74a9-4bf5-aa2e-7a09c3572e2b service nova] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:42:13.986438 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1484/2980] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:42:13 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 62 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:42:16.879757 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61469e99-09e5-4899-9a6a-8e06885eac43 service neutron] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:42:16.889432 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61469e99-09e5-4899-9a6a-8e06885eac43 service neutron] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:42:16.893978 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b072ed8e-5fee-4db4-8bb7-b04887984377 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:42:16.894578 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b072ed8e-5fee-4db4-8bb7-b04887984377 service neutron] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:42:16.895078 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b072ed8e-5fee-4db4-8bb7-b04887984377 service neutron] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:42:16.895078 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b072ed8e-5fee-4db4-8bb7-b04887984377 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:42:16.905766 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b072ed8e-5fee-4db4-8bb7-b04887984377 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:42:16.907208 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b072ed8e-5fee-4db4-8bb7-b04887984377 service neutron] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:42:16.916843 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1497/2981] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:42:16 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:42:23.970266 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=6 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:42:26.912825 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=6 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:44:40.350433 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26d877a1-74a9-4bf5-aa2e-7a09c3572e2b service nova] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:44:40.396011 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26d877a1-74a9-4bf5-aa2e-7a09c3572e2b service nova] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:44:40.401960 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d70e9ef0-f5a5-4468-bfd2-7fb010ac57ff service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2a3834fa6894dada3291e4781bbd553', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:44:40.402575 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d70e9ef0-f5a5-4468-bfd2-7fb010ac57ff service placement] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:44:40.402836 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d70e9ef0-f5a5-4468-bfd2-7fb010ac57ff service placement] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:44:40.403112 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d70e9ef0-f5a5-4468-bfd2-7fb010ac57ff service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:44:40.405216 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-d70e9ef0-f5a5-4468-bfd2-7fb010ac57ff service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:44:40.417233 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d70e9ef0-f5a5-4468-bfd2-7fb010ac57ff service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:44:40.418827 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d70e9ef0-f5a5-4468-bfd2-7fb010ac57ff service placement] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:44:40.428936 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1485/2982] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:44:40 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 79 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:44:46.344368 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b072ed8e-5fee-4db4-8bb7-b04887984377 service neutron] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:44:46.354226 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b072ed8e-5fee-4db4-8bb7-b04887984377 service neutron] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:44:46.357933 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cdd49e8-0acc-490e-afce-d4a110e8b546 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:44:46.358591 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cdd49e8-0acc-490e-afce-d4a110e8b546 service neutron] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:44:46.358866 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cdd49e8-0acc-490e-afce-d4a110e8b546 service neutron] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:44:46.359129 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cdd49e8-0acc-490e-afce-d4a110e8b546 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:44:46.360955 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-0cdd49e8-0acc-490e-afce-d4a110e8b546 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:44:46.372054 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cdd49e8-0acc-490e-afce-d4a110e8b546 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:44:46.374765 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cdd49e8-0acc-490e-afce-d4a110e8b546 service neutron] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:44:46.384197 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1498/2983] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:44:46 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:44:46.495071 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d70e9ef0-f5a5-4468-bfd2-7fb010ac57ff service placement] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:44:46.505700 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d70e9ef0-f5a5-4468-bfd2-7fb010ac57ff service placement] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:44:46.509967 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de860ea6-46ea-4731-ab53-4df4ced24ece service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:44:46.510568 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de860ea6-46ea-4731-ab53-4df4ced24ece service neutron] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:44:46.510865 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de860ea6-46ea-4731-ab53-4df4ced24ece service neutron] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:44:46.511099 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de860ea6-46ea-4731-ab53-4df4ced24ece service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:44:46.513153 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-de860ea6-46ea-4731-ab53-4df4ced24ece service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:44:46.527890 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de860ea6-46ea-4731-ab53-4df4ced24ece service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:44:46.529368 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de860ea6-46ea-4731-ab53-4df4ced24ece service neutron] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:44:46.540570 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1486/2984] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:44:46 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:44:56.380116 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=6 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:44:56.535543 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=22 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:46:32.378412 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac4db5d4-74b3-495c-8c3d-d8d498baad87 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:46:32.378917 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac4db5d4-74b3-495c-8c3d-d8d498baad87 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:46:32.378917 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac4db5d4-74b3-495c-8c3d-d8d498baad87 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:46:32.438046 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-ac4db5d4-74b3-495c-8c3d-d8d498baad87 None None] MFA Rules not processed for user `364a28c108e744728917ba2b0aa2347d`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:46:32.439316 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-ac4db5d4-74b3-495c-8c3d-d8d498baad87 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:46:32.466806 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1499/2985] 10.222.0.42 () {58 vars in 1062 bytes} [Sun Aug 28 01:46:32 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:46:32.475843 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de860ea6-46ea-4731-ab53-4df4ced24ece service neutron] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:46:32.506083 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de860ea6-46ea-4731-ab53-4df4ced24ece service neutron] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:46:32.510841 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-761073a7-52d0-4f18-9d55-e186b1e2b533 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:46:32.511632 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-761073a7-52d0-4f18-9d55-e186b1e2b533 service neutron] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:46:32.511959 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-761073a7-52d0-4f18-9d55-e186b1e2b533 service neutron] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:46:32.512247 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-761073a7-52d0-4f18-9d55-e186b1e2b533 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:46:32.539489 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-761073a7-52d0-4f18-9d55-e186b1e2b533 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:46:32.540940 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-761073a7-52d0-4f18-9d55-e186b1e2b533 service neutron] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:46:32.550039 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1487/2986] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:46:32 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 75 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:46:33.482472 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac4db5d4-74b3-495c-8c3d-d8d498baad87 None None] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:46:33.484856 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-ac4db5d4-74b3-495c-8c3d-d8d498baad87 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:46:33.535280 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac4db5d4-74b3-495c-8c3d-d8d498baad87 None None] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:46:33.542311 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16b59180-c02f-4b36-a97a-999154f61706 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '168b07df50c24593ba73dbde5b037aed', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'admin', 'reader', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:46:33.542972 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b59180-c02f-4b36-a97a-999154f61706 service nova] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:46:33.543211 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b59180-c02f-4b36-a97a-999154f61706 service nova] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:46:33.543481 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b59180-c02f-4b36-a97a-999154f61706 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:46:33.545335 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-16b59180-c02f-4b36-a97a-999154f61706 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:46:33.560235 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16b59180-c02f-4b36-a97a-999154f61706 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:46:33.561949 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16b59180-c02f-4b36-a97a-999154f61706 service nova] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:46:33.575094 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1500/2987] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:46:33 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Aug 28 01:46:33.660201 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-761073a7-52d0-4f18-9d55-e186b1e2b533 service neutron] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:46:33.669507 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-761073a7-52d0-4f18-9d55-e186b1e2b533 service neutron] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:46:33.673272 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d283c81-f5f3-4b5b-9db3-f1282e1c5c9c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:46:33.673942 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d283c81-f5f3-4b5b-9db3-f1282e1c5c9c service neutron] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:46:33.674243 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d283c81-f5f3-4b5b-9db3-f1282e1c5c9c service neutron] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:46:33.674512 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d283c81-f5f3-4b5b-9db3-f1282e1c5c9c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:46:33.676244 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-5d283c81-f5f3-4b5b-9db3-f1282e1c5c9c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:46:33.690550 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d283c81-f5f3-4b5b-9db3-f1282e1c5c9c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:46:33.692728 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d283c81-f5f3-4b5b-9db3-f1282e1c5c9c service neutron] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:46:33.704872 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1488/2988] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:46:33 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:46:34.899228 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16b59180-c02f-4b36-a97a-999154f61706 service nova] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:46:34.909615 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16b59180-c02f-4b36-a97a-999154f61706 service nova] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:46:34.913586 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cb2003e-3050-43f6-8291-e212fbf48482 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:46:34.914199 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cb2003e-3050-43f6-8291-e212fbf48482 service neutron] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:46:34.914478 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cb2003e-3050-43f6-8291-e212fbf48482 service neutron] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:46:34.914759 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cb2003e-3050-43f6-8291-e212fbf48482 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:46:34.916620 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-6cb2003e-3050-43f6-8291-e212fbf48482 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:46:34.931487 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cb2003e-3050-43f6-8291-e212fbf48482 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:46:34.933342 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cb2003e-3050-43f6-8291-e212fbf48482 service neutron] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:46:34.945456 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1501/2989] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:46:34 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 47 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:46:37.459969 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d283c81-f5f3-4b5b-9db3-f1282e1c5c9c service neutron] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:46:37.469244 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d283c81-f5f3-4b5b-9db3-f1282e1c5c9c service neutron] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:46:37.473788 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe87868f-141e-43dd-8b55-d681b83fb635 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '364a28c108e744728917ba2b0aa2347d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:46:37.474399 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe87868f-141e-43dd-8b55-d681b83fb635 service neutron] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:46:37.474743 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe87868f-141e-43dd-8b55-d681b83fb635 service neutron] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:46:37.475021 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe87868f-141e-43dd-8b55-d681b83fb635 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:46:37.550287 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe87868f-141e-43dd-8b55-d681b83fb635 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:46:37.552030 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe87868f-141e-43dd-8b55-d681b83fb635 service neutron] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:46:37.563085 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1489/2990] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:46:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:46:40.733365 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cb2003e-3050-43f6-8291-e212fbf48482 service neutron] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:46:40.735689 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-6cb2003e-3050-43f6-8291-e212fbf48482 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:46:40.755604 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cb2003e-3050-43f6-8291-e212fbf48482 service neutron] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:46:40.761153 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba563a37-3113-4d15-80b6-0cb7665b316f service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2a3834fa6894dada3291e4781bbd553', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:46:40.761778 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba563a37-3113-4d15-80b6-0cb7665b316f service placement] REQUEST_METHOD: `GET` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:46:40.762059 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba563a37-3113-4d15-80b6-0cb7665b316f service placement] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:46:40.762332 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba563a37-3113-4d15-80b6-0cb7665b316f service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:46:40.764295 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-ba563a37-3113-4d15-80b6-0cb7665b316f service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:46:40.777029 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba563a37-3113-4d15-80b6-0cb7665b316f service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:46:40.778695 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba563a37-3113-4d15-80b6-0cb7665b316f service placement] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:46:40.789584 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1502/2991] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:46:40 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 57 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:46:43.859256 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe87868f-141e-43dd-8b55-d681b83fb635 service neutron] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:46:43.869128 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe87868f-141e-43dd-8b55-d681b83fb635 service neutron] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:46:43.873905 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80248ab9-6584-4756-9df5-5eaf3376b013 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:46:43.874538 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80248ab9-6584-4756-9df5-5eaf3376b013 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:46:43.874825 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80248ab9-6584-4756-9df5-5eaf3376b013 admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:46:43.875103 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80248ab9-6584-4756-9df5-5eaf3376b013 admin admin] PATH_INFO: `/v3/users/41a51cf16450452194671aa46f4f06c4` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:46:43.876306 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80248ab9-6584-4756-9df5-5eaf3376b013 admin admin] RBAC: Authorizing `identity:delete_user(user_id=41a51cf16450452194671aa46f4f06c4)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:46:43.878264 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80248ab9-6584-4756-9df5-5eaf3376b013 admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:46:43.929183 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-80248ab9-6584-4756-9df5-5eaf3376b013 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '41a51cf16450452194671aa46f4f06c4'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:46:43.942656 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-80248ab9-6584-4756-9df5-5eaf3376b013 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '41a51cf16450452194671aa46f4f06c4'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:46:43.951532 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-80248ab9-6584-4756-9df5-5eaf3376b013 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '41a51cf16450452194671aa46f4f06c4'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:46:43.958383 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-80248ab9-6584-4756-9df5-5eaf3376b013 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '41a51cf16450452194671aa46f4f06c4'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:46:43.959011 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-80248ab9-6584-4756-9df5-5eaf3376b013 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '41a51cf16450452194671aa46f4f06c4'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:46:43.965062 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1490/2992] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 01:46:43 2022] DELETE /identity/v3/users/41a51cf16450452194671aa46f4f06c4 => generated 0 bytes in 106 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:46:44.148468 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba563a37-3113-4d15-80b6-0cb7665b316f service placement] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:46:44.179770 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba563a37-3113-4d15-80b6-0cb7665b316f service placement] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:46:44.252563 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccf4ce73-84f7-4341-9fc6-c4e9fc339734 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:46:44.253402 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccf4ce73-84f7-4341-9fc6-c4e9fc339734 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:46:44.253682 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccf4ce73-84f7-4341-9fc6-c4e9fc339734 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:46:44.254124 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccf4ce73-84f7-4341-9fc6-c4e9fc339734 admin admin] PATH_INFO: `/v3/projects/b4ead68269cb4e19befec4146c93686d` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:46:44.255350 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccf4ce73-84f7-4341-9fc6-c4e9fc339734 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b4ead68269cb4e19befec4146c93686d)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:46:44.257603 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccf4ce73-84f7-4341-9fc6-c4e9fc339734 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:46:44.304287 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:46:44.313727 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-ccf4ce73-84f7-4341-9fc6-c4e9fc339734 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b4ead68269cb4e19befec4146c93686d', 'request_id': 'req-ccf4ce73-84f7-4341-9fc6-c4e9fc339734'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:46:44.319898 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-ccf4ce73-84f7-4341-9fc6-c4e9fc339734 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b4ead68269cb4e19befec4146c93686d', 'request_id': 'req-ccf4ce73-84f7-4341-9fc6-c4e9fc339734'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:46:44.326467 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-ccf4ce73-84f7-4341-9fc6-c4e9fc339734 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b4ead68269cb4e19befec4146c93686d', 'request_id': 'req-ccf4ce73-84f7-4341-9fc6-c4e9fc339734'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:46:44.340732 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-ccf4ce73-84f7-4341-9fc6-c4e9fc339734 admin admin] The token cache is being invalidate because project b4ead68269cb4e19befec4146c93686d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81508) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 01:46:44.341062 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-ccf4ce73-84f7-4341-9fc6-c4e9fc339734 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:46:44.341724 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-ccf4ce73-84f7-4341-9fc6-c4e9fc339734 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:46:44.343236 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1503/2993] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 01:46:44 2022] DELETE /identity/v3/projects/b4ead68269cb4e19befec4146c93686d => generated 0 bytes in 195 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:46:44.347675 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80248ab9-6584-4756-9df5-5eaf3376b013 admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:46:44.349254 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-80248ab9-6584-4756-9df5-5eaf3376b013 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:46:44.363113 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80248ab9-6584-4756-9df5-5eaf3376b013 admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:46:44.367158 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:46:44.425964 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bef4806-9a1b-4771-b0c4-dd0bbc778cbd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:46:44.426627 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bef4806-9a1b-4771-b0c4-dd0bbc778cbd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:46:44.426903 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bef4806-9a1b-4771-b0c4-dd0bbc778cbd admin admin] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:46:44.427165 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bef4806-9a1b-4771-b0c4-dd0bbc778cbd admin admin] PATH_INFO: `/v3/users/e2091185562548218ab23c4797dcded8` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:46:44.428345 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bef4806-9a1b-4771-b0c4-dd0bbc778cbd admin admin] RBAC: Authorizing `identity:delete_user(user_id=e2091185562548218ab23c4797dcded8)` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:46:44.430369 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bef4806-9a1b-4771-b0c4-dd0bbc778cbd admin admin] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:46:44.489136 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-8bef4806-9a1b-4771-b0c4-dd0bbc778cbd admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e2091185562548218ab23c4797dcded8'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:46:44.502009 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-8bef4806-9a1b-4771-b0c4-dd0bbc778cbd admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e2091185562548218ab23c4797dcded8'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:46:44.510515 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-8bef4806-9a1b-4771-b0c4-dd0bbc778cbd admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e2091185562548218ab23c4797dcded8'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:46:44.517817 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-8bef4806-9a1b-4771-b0c4-dd0bbc778cbd admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e2091185562548218ab23c4797dcded8'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:46:44.518487 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.notifications [None req-8bef4806-9a1b-4771-b0c4-dd0bbc778cbd admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e2091185562548218ab23c4797dcded8'} {{(pid=81507) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:46:44.525192 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1491/2994] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Aug 28 01:46:44 2022] DELETE /identity/v3/users/e2091185562548218ab23c4797dcded8 => generated 0 bytes in 178 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:46:44.707910 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccf4ce73-84f7-4341-9fc6-c4e9fc339734 admin admin] Authenticating user token {{(pid=81508) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:46:44.709754 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-ccf4ce73-84f7-4341-9fc6-c4e9fc339734 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:46:44.726356 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccf4ce73-84f7-4341-9fc6-c4e9fc339734 admin admin] Validating token access rules against request {{(pid=81508) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:46:44.807672 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffe30436-e047-448e-9a2b-3d09cbb91299 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e5cb815a43fc4460b6ddf0175f087128', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cca4b1c3ff844aae827dd439e67820c7', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81508) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:46:44.808523 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffe30436-e047-448e-9a2b-3d09cbb91299 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:46:44.808786 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffe30436-e047-448e-9a2b-3d09cbb91299 admin admin] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:46:44.809109 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffe30436-e047-448e-9a2b-3d09cbb91299 admin admin] PATH_INFO: `/v3/projects/c1e59266819e4570885584a9bd3e031d` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:46:44.810930 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffe30436-e047-448e-9a2b-3d09cbb91299 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c1e59266819e4570885584a9bd3e031d)` {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:46:44.813770 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffe30436-e047-448e-9a2b-3d09cbb91299 admin admin] RBAC: Authorization granted {{(pid=81508) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:46:44.864452 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-ffe30436-e047-448e-9a2b-3d09cbb91299 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c1e59266819e4570885584a9bd3e031d', 'request_id': 'req-ffe30436-e047-448e-9a2b-3d09cbb91299'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:46:44.869960 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-ffe30436-e047-448e-9a2b-3d09cbb91299 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c1e59266819e4570885584a9bd3e031d', 'request_id': 'req-ffe30436-e047-448e-9a2b-3d09cbb91299'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:46:44.876254 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-ffe30436-e047-448e-9a2b-3d09cbb91299 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c1e59266819e4570885584a9bd3e031d', 'request_id': 'req-ffe30436-e047-448e-9a2b-3d09cbb91299'} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:46:44.890450 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-ffe30436-e047-448e-9a2b-3d09cbb91299 admin admin] The token cache is being invalidate because project c1e59266819e4570885584a9bd3e031d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81508) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Aug 28 01:46:44.890733 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-ffe30436-e047-448e-9a2b-3d09cbb91299 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:46:44.891364 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.notifications [None req-ffe30436-e047-448e-9a2b-3d09cbb91299 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81508) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Aug 28 01:46:44.892868 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1504/2995] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Aug 28 01:46:44 2022] DELETE /identity/v3/projects/c1e59266819e4570885584a9bd3e031d => generated 0 bytes in 186 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Aug 28 01:46:54.521023 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=50,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:46:54.887502 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=61,keystone:DELETE=11,keystone:INSERT=2 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:49:47.367103 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bef4806-9a1b-4771-b0c4-dd0bbc778cbd admin admin] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:49:47.368581 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-8bef4806-9a1b-4771-b0c4-dd0bbc778cbd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:49:47.456114 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bef4806-9a1b-4771-b0c4-dd0bbc778cbd admin admin] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:49:47.520446 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-623bb2f9-9706-4517-95f6-49b65bc76dd7 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2a3834fa6894dada3291e4781bbd553', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:49:47.521110 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-623bb2f9-9706-4517-95f6-49b65bc76dd7 service placement] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:49:47.521371 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-623bb2f9-9706-4517-95f6-49b65bc76dd7 service placement] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:49:47.521655 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-623bb2f9-9706-4517-95f6-49b65bc76dd7 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:49:47.523537 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-623bb2f9-9706-4517-95f6-49b65bc76dd7 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:49:47.535854 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-623bb2f9-9706-4517-95f6-49b65bc76dd7 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:49:47.537485 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-623bb2f9-9706-4517-95f6-49b65bc76dd7 service placement] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:49:47.548313 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1492/2996] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:49:47 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 182 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:49:57.543483 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=63 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:50:47.332947 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d77e6ee-51c3-4c35-9bed-93e0d466a565 None None] REQUEST_METHOD: `POST` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:50:47.334349 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d77e6ee-51c3-4c35-9bed-93e0d466a565 None None] SCRIPT_NAME: `/identity` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:50:47.334794 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d77e6ee-51c3-4c35-9bed-93e0d466a565 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81508) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:50:47.381750 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.auth.core [None req-8d77e6ee-51c3-4c35-9bed-93e0d466a565 None None] MFA Rules not processed for user `c2a3834fa6894dada3291e4781bbd553`. Rule list: `[]` (Enabled: `True`). {{(pid=81508) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Aug 28 01:50:47.382980 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG keystone.common.fernet_utils [None req-8d77e6ee-51c3-4c35-9bed-93e0d466a565 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81508) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:50:47.392448 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: [pid: 81508|app: 0|req: 1505/2997] 10.222.0.26 () {58 vars in 1015 bytes} [Sun Aug 28 01:50:47 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 61 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Aug 28 01:50:47.408806 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-623bb2f9-9706-4517-95f6-49b65bc76dd7 service placement] Authenticating user token {{(pid=81507) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Aug 28 01:50:47.410485 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.fernet_utils [None req-623bb2f9-9706-4517-95f6-49b65bc76dd7 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81507) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Aug 28 01:50:47.423924 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-623bb2f9-9706-4517-95f6-49b65bc76dd7 service placement] Validating token access rules against request {{(pid=81507) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 28 01:50:47.430140 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f70b1d01-563b-436e-9792-f23faf7f8aa9 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c2a3834fa6894dada3291e4781bbd553', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f53e6fdb369b480fbebb4086a505df67', 'project_domain_id': 'default', 'roles': ['service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81507) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 28 01:50:47.430757 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f70b1d01-563b-436e-9792-f23faf7f8aa9 service placement] REQUEST_METHOD: `GET` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 28 01:50:47.431025 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f70b1d01-563b-436e-9792-f23faf7f8aa9 service placement] SCRIPT_NAME: `/identity` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 28 01:50:47.431290 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f70b1d01-563b-436e-9792-f23faf7f8aa9 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81507) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 28 01:50:47.440677 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f70b1d01-563b-436e-9792-f23faf7f8aa9 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Aug 28 01:50:47.442417 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f70b1d01-563b-436e-9792-f23faf7f8aa9 service placement] RBAC: Authorization granted {{(pid=81507) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Aug 28 01:50:47.453370 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: [pid: 81507|app: 0|req: 1493/2998] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Aug 28 01:50:47 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Aug 28 01:50:57.379123 dv-cinder-master-3-iscsi devstack@keystone.service[81508]: DEBUG dbcounter [-] [81508] Writing DB stats keystone:SELECT=9 {{(pid=81508) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Aug 28 01:50:57.448756 dv-cinder-master-3-iscsi devstack@keystone.service[81507]: DEBUG dbcounter [-] [81507] Writing DB stats keystone:SELECT=6 {{(pid=81507) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}